Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model
Organizations: Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, IN 47907 USA
Abstract
Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer each action's consequences implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities, not generated text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions. Project page: https://ys-qu.github.io/jev-ad-website.
Figures & tables
| Simulator (KoMA) | NGSIM (recorded traffic) | |
|---|---|---|
| Other vehicles | IDM/MOBIL | recorded, reacting via IDM |
| Episode start | a KoMA scene | a recorded vehicle as the AV |
| AV driver | rule 50 %, Jev-AD 50 % | rule 50 %, Jev-AD 50 % |
| Perturbed episodes | 49 % | 50 % |
| Training episodes | 17,982 | 11,992 |
| States / labels | 510 k / 1.86 M | 209 k / 0.93 M |
| Jev (ours) | GPT-6 luna | GPT-4o mini | Llama-3 8B | |
| AUROC unsafe/safe | 1.000 | 0.999 | 1.000 | 0.930 |
| Unsafe blocked (of 23) | 100% | 100% | 100% | 87% |
| Safe blocked (of 105) | 0% | 4.1% | 0% | 1.0% |
| Borderline refused (of 10) | 0% | 37% | 20% | 0% |
| Intensity in range | 100% | 100% | 93.5% | 96.8% |
| Graded Spearman | 1.00 | 0.98 | 0.98 | 0.98 |
| Source | Danger recall | False alarm | Collision recall | Err. (m) |
|---|---|---|---|---|
| Simulator (KoMA) | ||||
| Constant velocity | 0.891 | 6.1% | 0.951 | 1.07 / 7.60 |
| Oracle (other AV idle) | 0.953 | 0.6% | 0.969 | – |
| Jev-AD world model (ours) | 0.946 | 2.9% | 0.978 | 0.29 / 3.65 |
| Replayed traffic (NGSIM, Sec. IV-G ) | ||||
| Constant velocity | 0.923 | 17.0% | 0.951 | 0.98 / 3.82 |
| (m/s) under command | |||||||
| Coll. | Compl. | hurry | none | gentle | stop | Succ. | |
| (%) | (%) | (%) | |||||
| KoMA agent ∗ | 33.7 | 62.0 | 25.3 | 20.1 | 17.6 | 10.6 | 50 |
| Jev + KoMA text | 26.0 | 73.8 | 25.8 | 20.1 | 8.8 | 3.8 | 60 |
| GPT-4o-mini + WM | 2.2 | 78.2 | 29.3 | 25.0 | 10.2 | 20.1 | 98 |
| GPT-6-luna + WM | 0.6 | 98.8 | 22.9 | 19.2 | 13.4 | 4.2 | 100 |
| Test | Metric | Jev-AD | GPT-6 luna † |
|---|---|---|---|
| Understanding ‡ | intent achieved | 92.9% | 91.6% |
| 26 commands | paraphrases (12) | 89.8% | 87.3% |
| 20 scenes | negations (7) | 97.5% | 98.6% |
| Chinese (3) | 90.8% | 86.7% | |
| unsafe (2) ∗ | 98.8% | 100% | |
| collisions | 0.6% | 0.6% |
| (m/s) under command | ||||||
|---|---|---|---|---|---|---|
| Coll. (%) | Compl. (%) | hurry | none | gentle | stop | |
| Jev + KoMA text | 48.2 | 51.8 | 11.2 | 6.8 | 2.3 | 0.5 |
| Jev-AD, const. vel. | 1.8 | 98.0 | 8.4 | 6.5 | 3.9 | 1.2 |
| Jev-AD, oracle | 0.0 | 100.0 | 8.9 | 6.6 | 3.7 | 0.5 |
| Jev-AD, WM (ours) | 2.8 | 97.0 | 8.5 | 6.4 | 3.6 | 0.6 |
Autonomous Driving
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
See all 574 papers