From Language Priors to Field Adaptation: Preference Learning for Traversability Estimation
Authors: Simon Schwaiger, David Seyser, Alessandro Scherl, Zlatan Ajanović, Wilfried Wöber, Gerald Steinbauer-Wagner
Organizations: Graz University of Technology, Institute of Software Engineering and Artificial Intelligence, Graz, Austria · University of Applied Sciences Technikum Wien, Research Group Automation Technology and Robotic Systems, Vienna, Austria · University of Alicante, Department of Computer Technology, Alicante, Spain · University of Natural Resources and Life Sciences, Institute for Integrative Nature Conservation Research, Vienna, Austria
Image-based traversability estimation is inherently dependent on the robot platform, deployment domain, and mission preferences, which limits the applicability of purpose-trained models. To facilitate domain adaptation, this work aims to reduce the number of required annotations in the target domain using sample-efficient preference learning. Our method represents traversability through von Mises-Fisher mixture prototypes in a frozen vision-language feature space. Relative natural-language rules provide a commonsense prior, while sparse relative image annotations adapt the prototype directions and utilities to a target domain through computationally and sample-efficient fine-tuning. Experiments on WayFAST demonstrate accuracy competitive with end-to-end trained estimators while enabling sample-efficient image-based adaptation. Qualitative experiments further demonstrate the language prior's zero shot applicability and the fine-tuned estimator's improved dense prediction on semantic maps. Evaluation is complemented via semantic interpretation of learned prototypes by dissecting semantically close natural language prompts. Code and trained estimators available at https://resireg.github.io
Figures & tables
Fig. 1: Language-Initialised Traversability Adaptation. In Stage 1, an LLM-generated vocabulary of relative traversability rules is embedded using a frozen language encoder to initialise the prompt prototypes. In Stage 2, the estimator is adapted to a new deployment domain using relative image annotations. This provides an initial commonsense traversability estimate while enabling lightweight and sample-efficient adaptation. Snowflake indicates a frozen VLM’s encoders.
Fig. 2: Method Overview. (a) relative traversability annotations in language and image modalities are encoded using a frozen VLM. (b) encoded annotation pairs are cached. A square hinge loss is applied over all cached annotations to parametrise a vMF mixture learning latent prototypes in VLM feature space using gradient descent. (c) The estimator builds a traversability estimate using learned per-prototype traversability t , prototype directions μ and shared concentration κ . (d) Dense traversability is estimated for each encoded image patch, or in the case of language-embedded pointclouds, for each point.
Fig. 3: Fine-Tuning Sample efficiency on WayFAST for language-prior and randomly initialised vMF prototypes, a parameter-matched MLP, as well as a parameter-matched transformer head, each with ≈8.2 k trainable parameters. All configurations are trained over 5 seeds on identical nested image subsets, plotting mean over the results. Shading indicates 95 % confidence interval. Dashed lines indicate the language prior. Our language-prior initialised estimator (blue line) is lower left compared to the other models, indicating improved fine-tuning sample-efficiency.
(a) Zero-Shot
HDR 0.1
HDR 0.25
HDR 0.5
(%) ↓
VLM + Static Prompt
23.9
34.4
35
Language Prior (Ours)
27.0
26.3
32.3
(b) Dataset Transfer
HDR 0.1
HDR 0.25
HDR 0.5
(%) ↓
Big Chungus [ 12 ]
19.9
16
27.4
Ours (random init)
29.1
19.0
27.8
Ours (language prior init)
30.9
19.9
24.11
TABLE I: WayFAST Traversability Estimation . Our method for estimating traversability in VLM feature space is competitive with purpose-trained end-to-end estimator networks, while minimising the amount of trained parameters. Results are highlighted as best , second-best , and third-best .
Fig. 4: Qualitative Traversability Estimation on RoboNav. Traversability predictions for outdoor scenes from the RoboNav Husky Route 1 recording. We compare the language prior to the language prior-initialised prototype bank adapted on the complete WayFAST training split. Predictions are shown for both image features and reconstructed 3D semantic maps using the same fixed [0,1] scale, where higher values indicate greater estimated traversability.
Fig. 5: Prototype semantics and traversability scores before and after image-based fine-tuning. Left: CLIP-Dissect descriptions inferred for each prototype direction μ from the language-prior and the fine-tuned estimator. Right: corresponding traversability scores tk , with gray circles denoting the text prior and blue squares the image-fine-tuned prototypes.
(a) Dataset Transfer
HDR 0.1
HDR 0.25
HDR 0.5
(%) ↓
K=4
29.3
19.0
27.4
K=8
27.8
18.3
27.2
K=12
30.1
19.6
27.6
K=16
29.1
19.0
27.8
K=24
29.6
19.3
27.5
K=32
29.1
19.0
27.2
TABLE II: vMF prototype-capacity ablation on WayFAST validation set. Results show HDR (%); lower is better.
Vocabulary
#Pairs
HDR 0.1
HDR 0.25
HDR 0.5
(%) ↓
tiny
307
27.0
26.3
32.3
small
521
31.1
30.5
34.8
medium
2,073
29.0
30.3
34.9
large
8,337
28.0
32.4
34.9
extra large
27,281
28.4
29.3
34.9
TABLE III: Language-prior vocabulary size ablation on WayFAST validation set. Results show HDR (%); lower is better.