Beyond the Camera: A Tri-Modal Driving World Model for Future RGB, Thermal, and LiDAR Forecasting in Adverse Weather
Authors: Faysal Mehrab Chowdhury, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Kishor Datta Gupta, Mohd Ariful Haque, Savannah Shannon
Organizations: Dept. of Computer Science and Engineering, North South University, Dhaka, Bangladesh · Dept. of Cyber Physical Systems, Clark Atlanta University, Atlanta, GA, USA · Dept. of Computer Science and Engineering, United International University, Dhaka, Bangladesh
Driving world models forecast what a vehicle's sensors will record next. Most forecast camera video and some add LiDAR, but thermal cameras, which still see in darkness and some adverse weather, are rarely a forecast target, and to our knowledge no world model forecasts all three sensors. We study joint forecasting of RGB, thermal, and LiDAR on real adverse-weather sequences of the DSERT-RoLL dataset. We post-train a pretrained video world model for each sensor and represent LiDAR as a depth video aligned with the RGB camera, with densification and encoding chosen by what the model's video tokenizer can reconstruct. A tri-sensor model sees all three sensors as one video. We score LiDAR in meters against measured returns and compare against persistence baselines passed through the same tokenizer as the model. With this matching, an apparent loss of the thermal model to persistence in fog disappears. On all 58 validation sequences in six weather conditions, post-trained RGB and thermal forecasts beat compression-matched persistence. Joint training lowers the LiDAR error at 2s by 2.1m at a small cost in camera fidelity, and only the tri-sensor model's LiDAR forecasts beat persistence in mean error. Blanking one sensor degrades the forecasts of the others more in snow than in fog.
Figures & tables
Figure 1: Forecasting beyond the camera. From five observed frames, our models forecast the next 2 s of RGB, thermal, and LiDAR depth in real adverse weather. Shown: the last forecast frame (2 s) of one fog window. Columns: real future, two samples of the tri-sensor model, and the single-sensor model. The tri-sensor LiDAR forecasts stay close to the real depth, while the single-sensor LiDAR forecast is distorted. The camera forecasts of the two models look similar; in all of them, the vehicle on the left differs from the real future and between samples.
Figure 2: Overview. (a) RGB and thermal frames are cropped and resized to 1280×704 videos; LiDAR sweeps become a depth video aligned with the RGB camera. (b) The single-sensor models forecast each video on its own; the tri-sensor model forecasts a 2×2 grid of all three sensors, with the fourth tile left black. (c) Cosmos-Predict2.5 with low-rank adapters forecasts 20 frames from 5 observed frames. (d) Tri-sensor forecasts are split back into their sensors, and every forecast is scored against the real future.
(a) Densification, one sequence, after the tokenizer
Input
PSNR ↑
SSIM ↑
Near med. (m) ↓
Raw points
15.6
0.498
–
Dilation
13.44
–
1.57
Dilation + closing (ours)
26.32
–
0.11
(b) Depth code, 12 windows, error (m) after the tokenizer
Code
Mean ↓
Median ↓
Near ↓
Far ↓
Table 1: The LiDAR depth video is designed against the tokenizer. PSNR in dB. (a) Without closing, holes survive densification and the tokenizer turns them into wrong depths. Measured on frames 5–29 of one Clear sequence. (b) Logarithmic and inverse-depth codes help near range but hurt overall and far range; near <10 m, far ≥30 m. Measured on 4 windows per evaluated weather of the initial split. – = not measured.
RGB (scene visible)
Thermal (scene visible)
LiDAR
Method
PSNR ↑
LPIPS ↓
F1 ↑
PSNR ↑
SSIM ↑
LPIPS ↓
MAE (m) ↓
δ1↑
Near (m) ↓
Obj. (m) ↓
Persistence
19.26
0.351
0.293
22.90
0.784
0.259
5.71
0.777
2.99
10.32
Persistence, compr.-matched
19.27
0.362
0.301
22.89
0.787
0.272
5.71
0.755
3.03
10.25
Single-sensor LoRA
21.79
0.284
0.447
24.86
0.811
0.243
6.81
0.752
7.12
8.22
Tri-sensor LoRA
21.71
0.304
0.410
24.61
0.810
0.270
4.68
0.794
3.70
7.21
Real future (upper bound)
37.56
0.102
0.607
44.25
0.980
0.055
1.25
0.944
0.31
–
Table 2: Forecasting on the full DSERT-RoLL validation set (58 sequences, six weather conditions, 300 windows), 2 s horizon. All rows are scored at the tile resolution of the tri-sensor model ( Sec. 4.3 ). The post-trained single-sensor models beat both persistence baselines on every camera metric. The tri-sensor camera forecasts are close to the single-sensor ones in PSNR and SSIM, but their thermal LPIPS is not better than persistence. For LiDAR, only the tri-sensor model beats persistence in mean error, while persistence stays best at near range. Camera metrics on scene-visible windows (RGB 285, thermal 280). Bold = best, underline = second best, excluding the real future. – = not applicable or not measured.
Model
Metric
Clear (30)
Fog (9)
L. Rain (5)
H. Rain (6)
L. Snow (6)
H. Snow (2)
Adverse (28)
All (58)
RGB
PSNR (dB) ↑
+2.94
+2.99
+2.27
− 0.17
+0.85
–
+1.60
+2.42
LPIPS ↓
− 0.097
− 0.052
− 0.085
− 0.046
− 0.034
–
− 0.056
− 0.081
Thermal
PSNR (dB) ↑
+2.74
− 0.97
+1.26
+0.04
+2.01
–
+0.57
+1.92
LPIPS ↓
− 0.048
+0.001
− 0.002
+0.008
− 0.012
–
− 0.002
− 0.031
LiDAR
MAE (m) ↓
+0.90
+4.85 ∗
− 0.61
− 0.35
− 0.69
+5.65
+1.42
+1.11 ∗
Tri-sensor LiDAR
MAE (m) ↓
− 0.68
− 1.46
− 1.46
− 1.29
− 1.34
− 2.43
− 1.50
− 1.03
Table 3: Gains over compression-matched persistence by weather condition. Paired difference (forecast minus compression-matched persistence), 2 s horizon, 95% cluster-bootstrap intervals over sequences. Single-sensor models at full resolution; the tri-sensor row is compared with persistence at tile resolution. Number of sequences in parentheses (L. = Light, H. = Heavy). Camera metrics on scene-visible windows; Heavy Snow has no scene-visible camera window. Bold = better, ∗ = worse, plain = no clear difference. Heavy Snow has two sequences, so no verdict is given.
Sensor
Metric
Tri − single
95% CI
LiDAR
MAE (m) ↓
−2.13
[−3.21,−1.25]
RMSE (m) ↓
−3.15
[−4.47,−2.02]
AbsRel ↓
−0.223
[−0.341,−0.128]
Near MAE (m) ↓
−3.42
[−5.35,−1.90]
δ1↑
+0.043
[+0.017,+0.071]
Chamfer (m) ↓
−0.13
[−0.30,−0.002]
Table 4: Joint training helps LiDAR; the camera forecasts change little. Paired differences (tri-sensor minus single-sensor), both at tile resolution, 2 s horizon, full validation set, 95% cluster-bootstrap intervals over sequences; camera metrics on scene-visible windows. All intervals exclude zero except for RGB PSNR.
Real
Tri-sensor
Single-sensor
(a) Observed frame (input)
RGB
Thermal
LiDAR
(b) 0.5 s after the last observed frame
RGB
Figure 3: A Light Snow window at night, a condition absent from training. One sample per model; in (a), model columns show the observed input as reproduced by each model. The forecasts of the two models are very close in all three sensors and keep the car ahead and the scene layout. In (b), the tri-sensor sample leaves more gaps than the real future in the near-range LiDAR depth (bottom of the image).
Sensor
Metric (2 s)
LoRA − zero-shot
95% CI
RGB
PSNR (dB) ↑
+2.17
[+0.84,+3.91]
LPIPS ↓
−0.042
[−0.069,−0.023]
Thermal
PSNR (dB) ↑
+8.94
[+5.77,+12.70]
LPIPS ↓
−0.107
[−0.130,−0.084]
LiDAR
MAE (m) ↓
−7.5
[−9.7,−5.1]
δ1↑
+0.243
[+0.187,+0.289]
Table 5: Post-training helps every sensor. Paired differences, all 90 windows of the initial split, 95% cluster-bootstrap intervals over sequences. All intervals exclude zero.
Blanked
Forecast
Metric
Fog (9 seq.)
Snow (4 seq.)
RGB
LiDAR
MAE (m)
+0.34 †
+4.47
RGB
Thermal
LPIPS
+0.012
+0.020
Thermal
RGB
LPIPS
+0.008
+0.089
Thermal
LiDAR
MAE (m)
+0.02 †
+0.20 †
LiDAR
RGB
LPIPS
+0.038
+0.203
LiDAR
Thermal
LPIPS
+0.012
+0.028
Table 6: Reliance grows from fog to snow. Degradation of a forecast when another sensor is blanked in the observed frames (positive = worse). Tri-sensor model, initial split, one sample per blanked run. Snow pools Light and Heavy Snow (2 sequences each), so snow values are indicative. † 95% interval includes zero.
Method
AbsRel
RMSE
Near
Mid
Far
Hole
Full resolution
Persistence
0.379
13.65
3.47
5.56
12.13
0.058
Persist., matched
0.379
13.30
3.45
5.58
12.14
0.049
Single LoRA
0.582
14.22
7.61
5.73
10.75
0.042
Tile resolution
Persistence
0.349
12.99
2.99
5.34
12.13
0.052
Table 7: Further LiDAR depth metrics , full validation, 2 s horizon. AbsRel and RMSE over all measured points; MAE (m) per distance band; hole = share of measured points whose forecast pixel is empty. Lower is better for all columns; matched = compression-matched. Bold = best at tile resolution.