Organizations: School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China · Southeast University Nanjing Jiangbei New Area Innovation Research Institute, Nanjing, China · College of Intelligence Science and Technology, and the National Key Laboratory of Equipment State Sensing and Smart Support, National University of Defense Technology, Changsha 410008, China · School of Transportation, Shandong University of Science and Technology, Qingdao 266590, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Jetour Auto, a Chery Auto company, and also with the Data & Intelligence Center, a division of Chery Auto company, Anhui 241000, China
Explainability and transparent decision-making are essential for the safe deployment of autonomous driving systems. Scene captioning summarizes environmental conditions and risk factors in natural language, improving transparency, safety, and human--robot interaction. However, most existing approaches target structured urban scenarios; in off-road environments, they are vulnerable to single-modality degradations caused by rain, fog, snow, and darkness, and they lack a unified framework that jointly models structured scene captioning and path planning. To bridge this gap, we propose Wild-Drive, an efficient framework for off-road scene captioning and path planning. Wild-Drive adopts modern multimodal encoders and introduces a task-conditioned modality-routing bridge, MoRo-Former, to adaptively aggregate reliable information under degraded sensing. It then integrates an efficient large language model (LLM), together with a planning token and a gate recurrent unit (GRU) decoder, to generate structured captions and predict future trajectories. We also build the OR-C2P Benchmark, which covers structured off-road scene captioning and path planning under diverse sensor corruption conditions. Experiments on OR-C2P dataset and a self-collected dataset show that Wild-Drive outperforms prior LLM-based methods and remains more stable under degraded sensing. The code and benchmark will be publicly available at https://github.com/wangzihanggg/Wild-Drive.
Figures & tables
Fig. 1: Wild-Drive unifies camera-LiDAR as input for off-road scene structured captioning and path planning, and provides adaptability to sensor corruption.
Fig. 2: The overview of our proposed Wild-Drive. It fuses camera–LiDAR features into multimodal tokens and uses MoRo-Former for query-conditioned routing and token compression. An LLM generates scene captions and planning tokens, which a GRU-based planner decodes into multimodal trajectories.
Fig. 3: The MoRo-Former module. It performs locality-aware masked attention to predict modality probabilities and hard-route task queries to LiDAR, camera, or fusion experts. The routed features are aggregated and compressed into compact task tokens for the LLM.
Dataset Name
OR-C2P
Weather Types
5
Illumination Types
4
Terrain Types
10
Frames with Obstacles
19,527
Train Sequences / Frames
100 / 39,727
Validation Sequences / Frames
15 / 5,717
TABLE I: Detailed Data Types and Data Split of the OR-C2P Dataset
Fig. 4: Experimental setup and data collection route map of the self-collected dataset
Fig. 5: Quantitative analysis of Wild-Drive for scene captioning and path planning on the OR-C2P dataset. In the path planning visualizations, red indicates the model-predicted trajectory and green indicates the ground truth trajectory.
Approach
Modality
OR-C2P Dataset
SC Dataset
Params
B-1 ↑
B-2 ↑
B-4 ↑
BERT-P ↑
B-1 ↑
BERT-P ↑
Mini-GPT4 [ 35 ]
V
31.27
16.57
5.52
77.43
29.52
74.26
7B
LLaVA1.5 [ 36 ]
V
36.75
18.25
9.71
81.34
33.27
71.55
7.06B
Instruct-BLIP [ 29 ]
V
37.59
19.75
10.29
82.89
33.41
70.52
7.91B
LLama-AdapterV2 [ 37 ]
V
49.17
38.46
27.29
81.61
46.33
79.92
7.22B
LiDAR-LLM [ 9 ]
L
68.04
58.61
46.11
87.49
66.19
88.47
6.96B
TABLE II: Comprehensive Evaluation of Off-Road Scene Captioning Performance on the OR-C2P and SC Datasets (B-n represents BLEU-n; BERT-P represents BERT performance).
Approach
Method Type
FDE ↓
minADE ↓
CoverNet [ 38 ]
Classif.-based
2.47
1.31
MTP [ 39 ]
Reg.-based
1.59
0.84
MultiPath [ 40 ]
Anchor-based
1.56
0.92
TopoPath [ 20 ]
Transf.-based
0.92
0.43
BEV-LLM-GRU [ 10 ]
LLM-based
1.33
0.97
Wild-Drive (Ours)
LLM-based
1.09
0.66
TABLE III: Comprehensive Evaluation of Off-Road Path Planning Performance on the OR-C2P Datasets.
Approach
B-1 ↑
B-4 ↑
BERT-P ↑
Wild-Drive (Ours)
71.72
49.26
98.13
w/ DINOv3 & Q-F
63.41
44.59
86.24
w/ V.Net & Q-F
60.39
41.72
83.51
w/ DINOv3 & V.Net & Q-F
66.79
45.92
92.74
TABLE IV: Ablation study results for Wild-Drive evaluating the contribution of each key component on the OR-C2P dataset.
Robotics, Khalifa University, Abu Dhabi, United Arab Emirates · Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates · Computer and Information Engineering, Khalifa University, Abu Dhabi, United Arab Emirates