cs.ROSep 30, 2026
SaveDrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
Abstract
Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.
Figures & tables
Figure 1: A frontier model driving a car through a fixed cone course. (1) Camera frame returned by observe() . (2) the model calls the MCP to control the car. (3) evaluation trajectory through the fixed cone course.
Figure 2: DrivingBench System. The agent applications run on a laptop (Codex, Claude Code, Cursor) and have access to our DrivingBench MCP server. The laptop used for evaluation and the comma four computer are both connected to the same phone hotspot. The comma device runs a modified version of the openpilot software, allowing the MCP to observe the front-facing camera and issue steering commands to the car.
| Initial interfaces (paths) | Final interface | |
|---|---|---|
| Model output | route geometry (waypoints, image-pixel paths or curvature segments) | steering %, speed, duration |
| Tools | observe , submit_path , wait_held , final_stop | observe , set_motion , stop_now |
| Harness owns | path tracking, accepting commands (as valid), turn shaping, endpoint braking | native openpilot limits only |
| Latency | absorbed: the car moves along the route while the model plans, no explicit exposure to the models | explicit: commands only start when accepted, and every response is timestamped |
| New command behavior | gets merged with the rest of the old existing/active path | replaces the running command, and expiration brakes |
| Infeasible/invalid request | rejected (explicitly) | clamped by default, only speeds above the ceiling are rejected |
Table 1: Initial (work-in-progress) and final action interfaces.
Figure 3: GPS trajectories of all 11 attempts on the cone course. Color identifies the model and line style the attempt within one chat. Markers show where the safety driver braked or where the car finished. The letters mark seven course sections (turns, bends, straights). 8 of the 11 attempts ended before section B, failing the first turn. The centerline was inferred from visible cones. Cone positions are illustrative (estimated from drone footage and satellite maps), rather than precisely surveyed. The reference used for progress/benchmark percents (Section 4.3 ) is the centerline up to the finish zone, which is 127 m long.
| Progress per attempt (%) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | Harness | 1 | 2 | 3 | Best (%) | Finish time | Tokens | Cost ($) |
| GPT-6 Astra | Codex | 49 | 100 | – | 100 | 5:22 | 7.8M | 9.75 |
| Claude Fable 5.1 | Claude Code | 9 | 10 | 45 | 45 | DNF | 3.6M | 3.95 |
| Grok 4.6 | Cursor | 8 | 11 | 10 | 11 | DNF | 1.0M | 0.65 |
| GPT-5.6 Sol | Codex | 6 | 6 | 6 | 6 | DNF | 1.6M | 1.05 |
Table 2: Results. Progress (%) per attempt; bold marks the first attempt that reached the best score for that model. Finish time gives the time between the first accepted command and the end of the last engagement. Tokens and cost are summed over all attempts (including reflections) at the public list prices as of September 2026. All models here ran with medium reasoning effort.
Figure 4: What the models saw when they failed. This figure includes one failed attempt per model. On the left, we see the last (narrow-camera) frame the model got before its final set_motion command. On the right, we see the model’s final command along with its stated reason , and what the outcome was.
Figure 5: Refusals during development (all GPT-6 Astra, using earlier interfaces or prompts). The excerpts are verbatim, and […] marks omissions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | # | Progress (%) | End of attempt |
|---|---|---|---|
| GPT-6 Astra | 1 | 49 | Operator brake: drifted onto the left cone line and island at the right turn into the cross aisle (D) |
| 2 | 100 | Finished: the model’s own stop_now inside the finish zone | |
| Claude Fable 5.1 | 1 | 9 | Operator brake: swung left across the cone line toward the median curb (A) |
| 2 | 10 | Operator brake: outside the cone line again, pointed at the median island’s curb (A) | |
| 3 | 45 | Operator brake: too little room for the right turn at D, pointed at the planter island | |
| Grok 4.6 | 1 | 8 | Operator brake: drove straight into the mini-cone gate toward the landscaped island (A) |
Table 3: How each attempt ended , from the road video and human operator notes. Letters are the course sections of Figure 3 .
Figure 6: Trace viewer. Trajectories recorded with our driving harness can be visualized in our trace viewer tool. It renders the recorded video of the evaluation run along with MCP tools that were used by the model and car telemetry such as steering, acceleration and position. All these data points are synchronized by timestamps and can be replayed alongside the video recording.
| Model | # | Prog. (%) | Dist. (m) | Dur. (s) | Cmds | Cmd/min | Moving (%) | Repl. | Med. gap (s) | Peak (m/s) |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 1 | 49 | 67.3 | 78 | 8 | 6.2 | 85 | 2/7 | 10.0 | 1.8 |
| 2 | 100 | 134.7 | 322 | 24 | 4.5 | 65 | 2/23 | 12.1 | 1.6 | |
| Claude Fable 5.1 | 1 | 9 | 17.5 | 41 | 3 | 4.4 | 45 | 0/2 | 18.9 | 1.8 |
| 2 | 10 | 27.3 | 190 | 4 | 1.3 | 15 | 0/3 | 60.8 | 1.9 | |
| 3 | 45 | 73.7 | 260 | 8 | 1.8 | 27 | 0/7 | 32.7 | 2.2 | |
| Grok 4.6 | 1 | 8 | 14.4 | 35 | 2 | 3.5 | 35 | 0/1 | 28.7 | 2.0 |
Table 4: Per-attempt statistics. Dur.: first accepted command to the end of the last engagement. Moving: share of that window with speed m/s. Repl.: commands sent while the previous one still had time left, out of the commands that followed another command. Med. gap: median time between consecutive commands. Peak: highest measured speed.
Figure 7: Course progress. Progress is the furthest point along the intended centerline (127 m, up to the finish zone) that the car reached while staying within 4 m of it. (a) Progress per attempt. Later attempts happen in the same chat after a fixed reflection prompt. (b) Progress over time, measured from the first accepted command. Horizontal guides mark the section labels in Figure 3 .
Figure 8: Acting on stale observations. For every accepted set_motion : (a) time since the previous tool call, mostly model deliberation (log scale); (b) distance the car drove between the capture of the last camera frame the model had received and the command’s acceptance. Bars are medians.
Figure 9: Requested versus actual steering. (a) Wheel-angle response to every command asking for a change of at least , normalised to the achievable change (request clipped to the EPS plateau). Thick lines are medians. (b) Delays after a command is accepted. Of the 32 commands in (a), 26 arrived after the previous command had expired and the car had braked. The EPS only turns the wheel once the car rolls, so from rest the wheel reaches 90% of its change after a median of 4.0 s (1.9 s when already rolling). (c) Achieved versus requested wheel angle per command. Requests of 100% ( ) reach a median of , so the upper of the command range wasn’t reached at the conditions of the model benchmark evaluations. (d) Measured path curvature from GPS against measured wheel angle, compared with a kinematic bicycle model. A 13.9:1 ratio fits better than the 18:1 in the harness documentation. At the plateau the turning radius is m, which implies roughly 25 s for a turn at 1 m/s, which is faster than the s we measured in a single sustained calibration run before the trial from rest and was used in the prompt’s example.
Figure 10: What the models commanded. (a) Distribution of the requested steering_percent for each attempt ( = number of commands). (b) Time between consecutive accepted set_motion calls, pooled over attempts (log scale).
Figure 11: Control traces, GPT-6 Astra. For each attempt: requested (black) and measured speed and steering-wheel angle over time. Shading marks periods with no active command, when the previous command had expired and the car was braked while the model deliberated. The dashed lines mark the typical EPS plateau.
Figure 12: Control traces, Claude Fable 5.1. As Figure 11 .
Figure 13: Control traces, Grok 4.6. As Figure 11 .
Figure 14: Control traces, GPT-5.6 Sol. As Figure 11 .