cs.ROOct 16, 2026

Structured Perception Representations for Off-road Dynamics Learning

Authors: Micah Nye, Samuel Triest, Matthew Sivaprakasam, Isaiah Adu, Sebastian Scherer, Wenshan Wang

Organizations: The Robotics Institute, Carnegie Mellon University · Pennsylvania State University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomously navigating off-road terrain at high speed demands accurate control, which often relies on dynamics models that capture complex interactions between the robot and terrain. In off-road regimes, terrain interaction is a major component of the vehicle's dynamics. Prior work attempts to capture this effect by training learned models of the vehicle's dynamics, conditioned on its perception of the local environment. However, said conditioning is often overly simplistic, ignoring the spatial structure of the dynamics problem. Our approach, SpatialBEV, exploits this spatial structure. At each step, we query the bird's-eye-view (BEV) features around the predicted state, conditioning the next prediction on current geometric and semantic cues. We additionally present a benchmarking approach that mines a dataset's most difficult scenarios. On datasets collected across seasons, SpatialBEV reduces prediction error by 17.1%, compared to physics-based and learned baselines. We further evaluate on hardware on high-speed (≥\geq10 m/s) and geometrically challenging courses with a 25.3% reduction in maximum cross-track error. More details can be found at https://theairlab.org/spatial-bev-dynamics .

Figures & tables

Figure 2: An overview of our dynamics model. We query each prior or predicted position in the static, structured BEV map at each step. This provides geometric and semantic context to our model for more accurate dynamics rollouts. Figure 3 details the structured representation formally.
Figure 3: Recurrent BEV Cells. At each step, the next predicted state determines a footprint over the BEV map. Sampled features are passed into the subsequent cell.
Figure 4: Similarly clustered actions yield different trajectories. Perception highlights how terradynamics are causing this difference. (C) travels further, going down a hill, as seen by the terrain map, (B) moves the least on flat dirt, while (A) travels further on flat, low-friction ice.
Figure 5: Vanilla BEV activations and our queries with reference BEV map and ground truth trajectory (white). Vanilla BEV’s activations fall on irrelevant regions, contrary to our queries along our prediction near the ground truth.
Figure 6: Offline full state prediction error distributions across 3 datasets. The Snow dataset is most challenging for all methods. Dense maps reduce error mean and distribution variance. Across all datasets, our method outperforms the baselines.
Table 2: Ablations of the correction step (top) and footprint (bottom) on all datasets.
Figure 7: Evaluation on the highest to lowest variance deciles of the clustered benchmark. Model performance increases with lower variance deciles.
Figure 8: Experiments for hardware evaluation on two challenging courses. Significant understeer can be seen by the KBM baseline leading to numerous interventions (A,D,E). Additionally all learned methods, including ours, experience cases of understeer like in the tight turn in C or G.
Table 3: Offline prediction results on closed-loop experiment data
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Offline qualitative results highlighting interesting prediction examples and environments.
Table 5: Extended offline position prediction results, including Transformer-based evaluations, evaluated across 3 datasets, alongside the full test set. Standard deviation removed for width-fitting.
 MeanStdMin5%Median95%Max
Forward vel. (m/s)4.661.87-0.121.924.408.2713.01
Lateral vel. (m/s)0.020.27-1.61-0.370.010.443.89
Slip (deg)0.274.41-38.43-6.100.137.2451.23
Yaw Rate (rad/s)-0.000.14-0.89-0.24-0.000.240.84
Pitch (deg)-0.073.84-18.92-7.340.126.7119.10
Roll (deg)0.223.31-18.85-5.280.375.2220.07
Appendix
Table 6: Statistics of entire dataset, encompassing test and train datasets
Figure 10: Perception and mapping flow diagram
Table 14
Figure 11: A robot following a target path with its optimal MPPI solution visualized. It is closely following the line in (A), highlighting accurate dynamics and control, but from (B) onwards, MPPI determines the optimal path undercuts the target path, nearly resulting in collision with dense vegetation.
HyperparameterVariableValue
LSTM hidden sizedhd_{h}128
Predictor input encoding dimdzpd_{z}^{p}16
Context input encoding dimdzcd_{z}^{c}144
BEV context latent sizedbd_{b}32
Spatial query points (tire track)F\mathcal{F}40
FPN CNN Downsample block–3×3\times
Appendix
Table 9: Hyperparameters for the LSTM architecture and perception encoders.
Figure 12: Tire track footprint visualization over a BEV map of the 100×100×0.4 m100\times 100\times 0.4\,\text{m} resolution aligned with the map (a) and rotated (b) to show how many new cells are queried. Vehicle outline is in black.
Figure 13: Footprints used for footprint ablation. Vehicle outline is in black.
Figure 14: Regions of interest for Vanilla BEV and our Spatial BEV approach on five different samples. Vanilla BEV attends to non-sensical regions for predicting motion, whereas our approach queries exactly where it predicts the robot to be, yielding a more informed environment context. The red line is the ground truth trajectory on all BEV plots. (A) through (E) show various environments and motions.
More in this topic

Robot Dynamics Learning

Robot dynamics learning builds models that predict how robots and objects move under actions, enabling control, planning, and adaptation.

See all 90 papers

Explore similar work

CardsList

  1. VertiAdaptor: Online Kinodynamics Adaptation for Vertically Challenging Terrain

    Mar 6, 2026Tong Xu, Chenhui Pan, Aniket Datar +1Dynamical SystemsOnline System Identification

    Department of Computer Science, George Mason University

  2. VertiAKD: Adaptive Off-Road Kinodynamics on Vertically Challenging Terrain

    Aug 2, 2026Tong Xu, Chenhui Pan, Francesco Cancelliere +1Robot Trajectory TrackingOnline System Identification

    Department of Computer Science, George Mason University, USA · Department of Electrical, Electronic and Computer Engineering, University of Catania, Italy

  3. NeSAM: Neuro-Symbolic Kinodynamics with Soil Adaptation for Off-Road Mobility

    Aug 21, 2026Chenhui Pan, Tong Xu, Francesco Cancelliere +1Terrain-Aware Robot LocomotionRobot Dynamics Learning

    Department of Computer Science, George Mason University, USA · Department of Electrical, Electronic and Computer Engineering, University of Catania, Italy