Traversability prediction is a core component of autonomous navigation, yet most vision-based methods operate frame-wise and produce temporally inconsistent predictions. We present SAM-VTP (Video Traversability Prediction), which fine-tunes SAM2's video memory pathway for dense, prompt-free traversability segmentation, conditioning each frame on a persistent memory of prior observations. Rather than inferring terrain geometry from appearance, we measure it: slope, roughness and height are computed from depth and fused into the image features. We also introduce an offline continuous supervision pipeline that generates dense per-pixel traversability targets in [0,1] from depth-derived geometry and semantic priors, requiring no robot deployment and no per-pixel annotation, capturing conditional feasibility that binary labels cannot represent. On TartanGround, a large-scale synthetic ground-robot dataset, SAM-VTP improves fuzzy Jaccard from 0.908 to 0.924 and binary IoU from 0.854 to 0.886 over SAM-TP, a frame-wise SAM2 baseline, against a seed-to-seed spread below 0.001, and cuts frame-to-frame flicker by 31%. In closed-loop trials at two field sites, driving the same planner and controller, the baseline's dropouts leave the planner without a feasible path 3.3x more often on a paved course and 6.9x on rough ground, while SAM-VTP runs at 21.4 FPS on the Orin Nano the robot carries.
Figures & tables
Fig. 1: Temporal comparison of SAM-TP (top), the frame-wise SAM2 traversability model released with GeNIE [ 1 ] and the closest published system to ours, against SAM-VTP (bottom) on consecutive frames from TartanGround ConstructionSite. At t2 SAM-TP loses roughly half of the traversable surface and recovers it at t3 , while SAM-VTP holds a coherent region across all three frames (fuzzy Jaccard, fJacc, 0.69 against 0.92 on this window). SAM-VTP’s output is also traversability-scaled, whereas SAM-TP’s is dominated by binary confidence.
Fig. 2: SAM-VTP: SAM2’s Hiera-T encoder and memory pathway. The four starred blocks are new or modified in this work: a depth-derived geometry encoder summed into the image features, a learned prompt replacing SAM2’s prompt encoder, a decoder emitting a single mask token, and a residual head refining the output at full resolution. Parameter counts are per block; the memory bank holds activations, not parameters.
Fig. 3: Continuous traversability supervision on TartanGround. Left to right: the semantic labels for one frame, the geometry-derived height above ground and local surface slope, and the final target y∈[0,1] . The target combines a platform-grounded semantic prior with continuous geometric modulation, and cannot be trivially derived from semantics or geometry alone.
Accuracy
Temporal
Throughput (FPS) ↑
Model
fJacc ↑
MAE ↓
ECE ↓
IoU ↑
Stab. ↓
Flick. ↓
RTX 5090
AGX Orin
Orin Nano
WayFASTER
0.791
0.066
0.039
0.740
0.0630
0.0264
456
52.8
26.8
WVN
0.760
0.079
0.048
0.741
0.0597
0.0246
19.5
– ‡
– ‡
SAM-TP
0.908
0.024
0.013
0.854
0.0387
0.0175
540*
44.9*
33.5*
SAM-VTP (ours)
0.924
0.018
0.010
0.886
0.0323
0.0120
296
30.1
21.4
SAM-TP (GeNIE) †
0.667
0.126
0.111
0.678
0.0908
0.0452
45.7
6.3
3.6
TABLE I: Accuracy, temporal consistency, and throughput on held-out TartanGround test trajectories. All models fine-tuned on TartanGround unless noted. Best in bold . fJacc is the fuzzy (Ruzicka) Jaccard index over continuous scores and is the primary metric; IoU is the binary Jaccard index at a per-model threshold calibrated on validation; ECE is the expected calibration error against the continuous targets. Stab. and Flick. are the mean per-sequence standard deviation of fJacc across frames and its mean absolute frame-to-frame change, both lower-is-better; basing them on fJacc rather than a thresholded score keeps them on the primary metric. Throughput uses each model’s own deployment resolution ( 512×512 except WayFASTER, which predicts a 100×100 BEV grid from 320×180 input; with targets interpolated to that grid and predictions upsampled back, limiting its attainable fidelity).
Component
Configuration
Mem.
Geo.
fJacc ↑
MAE ↓
ECE ↓
Flick. ↓
SAM-TP (frame-wise)
–
–
0.875
0.033
0.019
0.022
+ geometry
–
✓
0.900
0.026
0.014
0.019
SAM-VTP (memory)
✓
–
0.894
0.028
0.016
0.015
+ geometry (full)
✓
✓
0.914
0.022
0.013
0.013
Full, depth withheld
✓
–
0.853
0.042
0.031
0.020
TABLE II: Ablation on held-out TartanGround test trajectories, scored over 1000 sequence windows (24k frames). The first four rows form a 2×2 factorial over the two proposed components; SAM-TP + geometry is the fairness control, giving the frame-wise baseline the same depth-derived input as SAM-VTP. The last two rows withhold depth at inference and replace continuous targets with binary ones. Training budgets here are shorter than in Table I , so absolute scores are lower and the two tables are not directly comparable. Best in bold .
Fig. 4: Per-environment difference in fJacc between SAM-VTP and SAM-TP on the 22 held-out test trajectories, sorted by magnitude; bars are the mean over three seeds and error bars their standard deviation. SAM-VTP is ahead in 21 of 22 environments. The grey column gives SAM-TP’s own fJacc, showing that the largest gains fall where the frame-wise baseline is weakest ( r=−0.61 ) while environments it already scores above 0.95 gain least. OldScandinavia is clipped at the axis limit; its value is +0.141 .
Paved course
Rough terrain
Measure
TP
VTP
Ratio
TP
VTP
Ratio
No-path (%) ↓
32.4
9.9
0.31 ×
20.6
3.0
0.15 ×
Traversable frac. ↑
0.225
0.338
1.50 ×
0.076
0.216
2.86 ×
Flicker (region) ↓
0.129
0.030
0.23 ×
0.055
0.036
0.64 ×
Null predictions (%) ↓
7.9
0.9
0.12 ×
4.2
0.8
0.18 ×
Progress rate (m/s) ↑
0.141
0.182
1.29 ×
0.159
0.180
1.13 ×
TABLE III: Closed-loop navigation on a physical robot at two field sites: a paved course with mixed surfaces and vegetation (15 runs), and rough, undulating, overgrown ground (21 runs). The baseline is SAM-TP trained on the same data, objective and splits, with planner and controller shared and trials interleaved between models at both sites. SAM-TP takes RGB only and SAM-VTP takes RGB-D, so the table compares the two systems and does not separate memory from depth. Flicker is the mean absolute frame-to-frame change in predicted probability over the predicted region, the union of P>0.5 across the pair, as are traversable fraction and null predictions; both arms select P>0.5 independently on validation, and the ordering holds for any threshold in [0.1,0.8] . Every row separates completely on the course; on rough ground no-path, traversable fraction and flicker separate in all ten paired trials, a sign test at its floor ( p=0.002 ), while progress rate and distance overlap ( 8/10 and 7/10 pairs).
Fig. 5: Both models replayed over the same recorded frames in the deployed configuration (SAM-TP on RGB, SAM-VTP on RGB-D), so both see the same scene. Overlay opacity is the predicted probability; the white contour marks P>0.5 . The top two rows are the paved course, where SAM-TP breaks into disconnected fragments around clutter (2% against 32%) and again at the terrain and lighting transition at which its trials ended, where the planner was reporting no feasible path (2% against 28%); SAM-VTP holds a single coherent region in both. The bottom row is rough terrain, where SAM-TP under-segments a clearly defined path (6% against 25%) rather than fragmenting.