Autonomously navigating off-road terrain at high speed demands accurate control, which often relies on dynamics models that capture complex interactions between the robot and terrain. In off-road regimes, terrain interaction is a major component of the vehicle's dynamics. Prior work attempts to capture this effect by training learned models of the vehicle's dynamics, conditioned on its perception of the local environment. However, said conditioning is often overly simplistic, ignoring the spatial structure of the dynamics problem. Our approach, SpatialBEV, exploits this spatial structure. At each step, we query the bird's-eye-view (BEV) features around the predicted state, conditioning the next prediction on current geometric and semantic cues. We additionally present a benchmarking approach that mines a dataset's most difficult scenarios. On datasets collected across seasons, SpatialBEV reduces prediction error by 17.1%, compared to physics-based and learned baselines. We further evaluate on hardware on high-speed (≥10 m/s) and geometrically challenging courses with a 25.3% reduction in maximum cross-track error. More details can be found at https://theairlab.org/spatial-bev-dynamics .
Figures & tables
Figure 2: An overview of our dynamics model. We query each prior or predicted position in the static, structured BEV map at each step. This provides geometric and semantic context to our model for more accurate dynamics rollouts. Figure 3 details the structured representation formally.
Figure 3: Recurrent BEV Cells. At each step, the next predicted state determines a footprint over the BEV map. Sampled features are passed into the subsequent cell.
Figure 4: Similarly clustered actions yield different trajectories. Perception highlights how terradynamics are causing this difference. (C) travels further, going down a hill, as seen by the terrain map, (B) moves the least on flat dirt, while (A) travels further on flat, low-friction ice.
Figure 5: Vanilla BEV activations and our queries with reference BEV map and ground truth trajectory (white). Vanilla BEV’s activations fall on irrelevant regions, contrary to our queries along our prediction near the ground truth.
Figure 6: Offline full state prediction error distributions across 3 datasets. The Snow dataset is most challenging for all methods. Dense maps reduce error mean and distribution variance. Across all datasets, our method outperforms the baselines.
Table 2: Ablations of the correction step (top) and footprint (bottom) on all datasets.
Figure 7: Evaluation on the highest to lowest variance deciles of the clustered benchmark. Model performance increases with lower variance deciles.
Figure 8: Experiments for hardware evaluation on two challenging courses. Significant understeer can be seen by the KBM baseline leading to numerous interventions (A,D,E). Additionally all learned methods, including ours, experience cases of understeer like in the tight turn in C or G.
Table 3: Offline prediction results on closed-loop experiment data
Table 5: Extended offline position prediction results, including Transformer-based evaluations, evaluated across 3 datasets, alongside the full test set. Standard deviation removed for width-fitting.
Mean
Std
Min
5%
Median
95%
Max
Forward vel. (m/s)
4.66
1.87
-0.12
1.92
4.40
8.27
13.01
Lateral vel. (m/s)
0.02
0.27
-1.61
-0.37
0.01
0.44
3.89
Slip (deg)
0.27
4.41
-38.43
-6.10
0.13
7.24
51.23
Yaw Rate (rad/s)
-0.00
0.14
-0.89
-0.24
-0.00
0.24
0.84
Pitch (deg)
-0.07
3.84
-18.92
-7.34
0.12
6.71
19.10
Roll (deg)
0.22
3.31
-18.85
-5.28
0.37
5.22
20.07
Appendix
Table 6: Statistics of entire dataset, encompassing test and train datasets
Figure 10: Perception and mapping flow diagram
Table 14
Figure 11: A robot following a target path with its optimal MPPI solution visualized. It is closely following the line in (A), highlighting accurate dynamics and control, but from (B) onwards, MPPI determines the optimal path undercuts the target path, nearly resulting in collision with dense vegetation.
Hyperparameter
Variable
Value
LSTM hidden size
dh
128
Predictor input encoding dim
dzp
16
Context input encoding dim
dzc
144
BEV context latent size
db
32
Spatial query points (tire track)
F
40
FPN CNN Downsample block
–
3×
Appendix
Table 9: Hyperparameters for the LSTM architecture and perception encoders.
Figure 12: Tire track footprint visualization over a BEV map of the 100×100×0.4m resolution aligned with the map (a) and rotated (b) to show how many new cells are queried. Vehicle outline is in black.
Figure 13: Footprints used for footprint ablation. Vehicle outline is in black.
Figure 14: Regions of interest for Vanilla BEV and our Spatial BEV approach on five different samples. Vanilla BEV attends to non-sensical regions for predicting motion, whereas our approach queries exactly where it predicts the robot to be, yielding a more informed environment context. The red line is the ground truth trajectory on all BEV plots. (A) through (E) show various environments and motions.
Autonomous driving in off-road environments presents significant challenges due to the dynamic and unpredictable nature of unstructured terrain. Traditional kinodynamic models often struggle to generalize across diverse geometric and semantic terrain types, underscoring the need for real-time adaptation to ensure safe and reliable navigation. We propose VertiAdaptor (VA), a novel online adaptation framework that efficiently integrates elevation with semantic embeddings to enable terrain-aware kinodynamic modeling and planning via function encoders. VA learns a kinodynamic space spanned by a set of neural ordinary differential equation basis functions, capturing complex vehicle-terrain interactions across varied environments. After offline training, the proposed approach can rapidly adapt to new, unseen environments by identifying kinodynamics in the learned space through a computationally efficient least-squares calculation. We evaluate VA within the Verti-Bench simulator, built on the Chrono multi-physics engine, and validate its performance both in simulation and on a physical Verti-4-Wheeler platform. Our results demonstrate that VA improves prediction accuracy by up to 23.9% and achieves a 5X faster adaptation time, advancing the robustness and reliability of autonomous robots in complex and evolving off-road environments.
Tong Xu, Chenhui Pan, Aniket Datar +1
Department of Computer Science, George Mason University
Off-road mobility requires autonomous mobile robots to generalize across heterogeneous vehicle fleets and continuously changing terrain conditions. Existing cross-vehicle adaptation approaches generally assume flat terrain, while terrain-aware kinodynamic models often require platform-specific data collection and retraining. To this end, we propose VertiAKD, a unified framework for transferring and adapting off-road kinodynamic knowledge across diverse vehicles on geometrically and semantically complex terrain simultaneously. VertiAKD learns a shared mobility representation that jointly encodes vehicle configurations, trajectory transitions, and local elevation and semantic terrain features. Given limited data from a novel vehicle operating on unseen terrain, VertiAKD identifies the most relevant mobility descriptors and transfers their knowledge to initialize a terrain-aware kinodynamic model via function encoders, which is then periodically refined online from streaming observations without gradient-based retraining. We evaluate VertiAKD in the Verti-Bench simulator, built on the Chrono multi-physics engine, and on five physical configurations of the Verti-4-Wheeler platform. With only one minute of new trajectory data and associated terrain features, VertiAKD reduces long-horizon prediction error by up to 34.52% over direct mobility descriptor transfer across diverse unseen vehicle configurations and 94.43% over competing baselines. We further demonstrate robust closed-loop trajectory tracking in both simulation and physical experiments, highlighting the effectiveness of terrain-aware cross-vehicle knowledge transfer for accurate modeling and reliable off-road navigation.
Tong Xu, Chenhui Pan, Francesco Cancelliere +1
Department of Computer Science, George Mason University, USA · Department of Electrical, Electronic and Computer Engineering, University of Catania, Italy
Accurate prediction of off-road vehicle motion over deformable terrain remains challenging because sinkage, slip, and traction vary with local soil conditions. Existing learning-based kinodynamic models directly approximate vehicle-terrain interactions from data but do not explicitly represent soil mechanics and offer limited physical interpretability. To address these limitations, we present NeSAM, a neuro-symbolic framework that combines differentiable Bekker-Wong terramechanics with learned terrain representations and a Transformer-based residual dynamics model for long-horizon, six degree-of-freedom kinodynamic prediction. The terramechanics component models soil-dependent interaction forces, while the residual model corrects discrepancies between the analytical prediction and the observed vehicle dynamics. NeSAM further estimates physically meaningful soil parameters from terrain observations and updates them online using an extended Kalman filter. We evaluate NeSAM in Verti-Bench, a simulator built on the Chrono multiphysics engine, and validate its performance on a physical Verti-4-Wheeler platform. NeSAM improves prediction accuracy by up to 30% in simulation and 29% on real-world data relative to the strongest compared baselines. When integrated with a close-loop navigation controller, NeSAM further improves traversal success rate through online soil adaptation while reduces Hausdorff distance to the reference trajectory by 69.4%, indicating improved trajectory tracking accuracy.
Chenhui Pan, Tong Xu, Francesco Cancelliere +1
Department of Computer Science, George Mason University, USA · Department of Electrical, Electronic and Computer Engineering, University of Catania, Italy