DriveMind: Mind-Evolving Belief Tracking for Closed-Loop Autonomous Driving

Yan Shi; Kehua Chen; Jiarui Qi; Yihong Tang; Haichen Liao; Yinhai Wang

cs.AIOct 23, 2026

DriveMind: Mind-Evolving Belief Tracking for Closed-Loop Autonomous Driving

Authors: Yan Shi, Kehua Chen, Jiarui Qi, Yihong Tang, Haichen Liao, Yinhai Wang

Organizations: University of Washington · McGill University · University of Michigan

Abstract

Recent multimodal large language models (MLLMs) can perceive a traffic scene, reason about it, and propose a driving action within a single model. However, existing MLLM-based drivers reason from scratch at each frame and commit to a single explanation, discarding earlier hypotheses about the scene. When the scene is ambiguous, the agent locks in on the wrong interpretation and acts on it before evidence accumulates. In this paper, we propose DriveMind, an MLLM-based driving framework that keeps several hypotheses about the scene alive at the same time and updates their weights as new frames arrive. Inspired by particle filtering, DriveMind propagates each hypothesis through a VLM, scores it against the current observation, and resamples to maintain diversity. On Bench2Drive, adding belief tracking to the same driving pipeline improves the Driving Score (DS) from 68.50 to 81.37 and the Success Rate (SR) from 41.36% to 54.09%. The largest gains occur on ambiguous scenarios such as unsignalized intersections (+20.0 DS) and oncoming-traffic interactions (+20.5 DS). Beyond the overall metrics, DriveMind also improves fine-grained interactive driving abilities such as merging, emergency braking, and give-way.

Figures & tables

Figure 1: From single-chain reasoning to mind-evolving belief. (a) Instance-Based Framework: one rationale per timestep. (“CoT” stands for “Chain of Thought”) (b) Mind-Evolving Framework: multiple hypotheses $\{h_{t}^{k}\}$ propagated over time; darker color indicates higher hypothesis weight.

Figure 2: DriveMind overview. (a) An asynchronous fast–slow architecture: a slow vision-language model (VLM) maintains a multi-hypothesis belief state, while a fast diffusion planner generates trajectories. (b) Belief tracking through hypothesis propagation and reweighting. (c) The belief state evolves as new observations arrive.

Figure 3: Belief dynamics across four scenarios. Foggy Road (red), Urban Intersection (orange), Night Accident (blue), Parking Exit (green). (a) Entropy

U_{t}

starts near

\log K

and decays as the scene becomes less ambiguous. (b) The dominant weight

\max_{k}w_{t}^{k}

climbs toward certainty in most scenarios; Night Accident plateaus at

{\sim}0.6

, reflecting persistent ambiguity. (c)

\text{ESS}_{t}

drops below the resampling threshold

\alpha K

once belief concentrates, except in Night Accident where diversity is maintained throughout. Per-step visualizations appear in Appendix E .

Figure 4: Belief evolution and weighted action selection. The ego vehicle approaches an intersection with a left-turn command and a cyclist near the crosswalk. DriveMind starts with four equally weighted hypotheses and concentrates belief on cautious behavior as the cyclist’s relevance becomes clear: deceleration receives the highest final weight ( $0.78$ ), with stopping retained as a fallback ( $0.22$ ).

Scenario Category	DS $\uparrow$		SR (%) $\uparrow$		$\Delta$ DS
Scenario Category	DriveMind w/o belief	DriveMind	DriveMind w/o belief	DriveMind	$\Delta$ DS
Unsignalized Intersection	54.4	74.4	16.7	30.0	+20.0
Opposite Direction	42.7	63.2	13.3	13.3	+20.5
Pedestrian Crossing	53.3	70.2	20.0	30.0	+16.9
Obstacle / Hazard	84.0	98.7	73.3	96.7	+14.7
Merging	72.0	85.6	50.0	70.0	+13.6

Table 2: Results across nine Bench2Drive scenario categories.

\Delta

DS is the gain over DriveMind w/o belief.

Scenario	DS $\uparrow$		SR (%) $\uparrow$		$\Delta$ DS
Scenario	Baseline	DriveMind	Baseline	DriveMind	$\Delta$ DS
Parking–Crossing Pedestrian	53.2	100.0	20	100	+46.8
Merger into Slow Traffic (V2)	44.0	79.2	0	60	+35.2
Hazard at Side Lane (Two Ways)	66.6	100.0	40	100	+33.4
Non-Signalized Junction Right Turn	49.7	81.6	20	40	+31.9
T-Junction	59.2	86.0	0	60	+26.8

Table 3: Results on six selected challenging Bench2Drive scenarios. Baseline denotes DriveMind w/o belief; SR is reported in percent.

Table 7

Component-level latency			System-level latency
Component	Mean (ms)	P95 (ms)	Setting	Mean (ms)
High Level Hypothesis Evolving ( $K{=}2$ )	74.0	77.9	Synchronous baseline	177.0
High Level Hypothesis Evolving ( $K{=}4$ )	120.0	125.4	Async refresh ( $s{=}2$ )	94.7
High Level Hypothesis Evolving ( $K{=}6$ )	165.5	167.6	Async refresh ( $s{=}4$ )	65.5
Low-level control	45.0	45.0	Async refresh ( $s{=}6$ )	57.1

Table 6: Runtime analysis of DriveMind on a single NVIDIA RTX A6000 GPU. Left: isolated component latency. Right: system-level latency under shared-GPU execution. P95 denotes the 95th-percentile latency.

s

denotes the number of fast control steps between two slow reasoner updates.

Figure 5: Effect of hypothesis count

K

on behavioral diversity. (a) Weighted action disagreement. (b) Maneuver categories with total weight at least

0.05

. (c) ESS after reweighting; dashed lines mark

K

. Curves show means across four scenarios, each averaged over eight reasoning steps; shaded bands and error bars show one standard deviation across scenario means.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Parameter	Value
VLM Learning Rate	$2\times 10^{-5}$	Diffusion Learning Rate	$1\times 10^{-4}$
VLM Batch Size	8	Diffusion Batch Size	256
Weight Decay (VLM)	0.0001	DDPM Steps	20
Max Sequence Length	2048	Weight Decay (Diffusion)	0.0005

Appendix

Table 7: Training hyperparameters for DriveMind.

Figure 6: Diversity maintenance through resampling and regeneration. When the effective sample size $ESS_{t}$ falls below the threshold $\alpha K$ , DriveMind resamples high-weight hypotheses and regenerates near-duplicated hypotheses to maintain semantic diversity while preserving consistency with the current observation.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Constant Velocity	68.0	57.8	50.0	100	19.4	20.6
Ego Status MLP	93.0	77.3	83.6	100	62.8	65.6
VADv2- $V_{8192}$ ( Jiang et al., 2026 )	97.2	89.1	91.6	100	76.0	80.9
DrivingGPT ( Chen et al., 2025 )	98.9	90.7	94.9	95.6	79.7	82.4
UniAD ( Hu et al., 2023 )	97.8	91.9	92.9	100	78.8	83.4
TransFuser ( Chitta et al., 2023 )	97.7	92.8	92.8	100	79.2	84.0

Appendix

Table 8: Driving performance on Navsim

Figure 7: Intersection uncertainty. DriveMind initially maintains multiple plausible actions under ambiguity. As the pedestrian remains near the crosswalk and wet-road conditions persist, the belief concentrates on cautious behavior, assigning final weights of $0.55$ to deceleration and $0.45$ to stopping, while go-straight and lane-change actions are down-weighted to nearly zero.

Figure 8: Foggy weather with pedestrian uncertainty. DriveMind handles a foggy and wet-road scenario where a pedestrian appears near the roadside. Early hypotheses include stopping for a potential hazard, decelerating due to low visibility, continuing straight, and lane changing. As the scene evolves, the belief concentrates on the deceleration hypothesis, reaching weight $0.99$ at the final timestep. This illustrates how belief tracking supports cautious control under degraded visibility and pedestrian uncertainty.

Figure 9: Stable parking exit behavior in a low-ambiguity scenario. DriveMind initially maintains multiple plausible hypotheses because of wet road conditions and a nearby vehicle. As the front vehicle moves farther ahead and the lane remains clear, DriveMind quickly concentrates its belief on the go-straight hypothesis. At the final timestep, the weighted action assigns $0.99$ to go-straight, while stopping, deceleration, and lane-change are down-weighted to nearly zero. This case shows that DriveMind does not remain overly cautious when the visual evidence supports safe forward progress.

Figure 10: Nighttime driving with accident uncertainty. The ego vehicle drives in a dark, wet, and low-visibility scene with a police vehicle ahead. DriveMind initially maintains multiple plausible hypotheses with nearly uniform weights, including stopping, decelerating, going straight, and lane changing. As new observations confirm persistent poor visibility and the presence of the emergency vehicle, the belief shifts toward conservative behavior, with the final weighted action assigning the highest probability to deceleration while still retaining smaller probabilities for stopping and going straight.

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 534 papers

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving
Feb 25, 2026Lingjun Zhang, Yujian Yuan, Changjie Wu +7Reasoning for VLMs VLMs for Autonomous Driving
Amap, Alibaba Group · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1
LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers
Mar 2, 2026Fabian Schmidt, Karol Fedurko, Markus Enzweiler +1Language-Conditioned Robot Planning Autonomous Driving Planning
Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.
Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
May 19, 2026Zhengqi Sun, Yiwen Sun, Boxuan Liu +3Language Model-Based Control World Models
Department of Information Management, Peking University, Beijing 100871, China · School of Intelligence Science and Technology, Peking University, Beijing 100871, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing 100080, China +3

Feb 25, 2026J/K move · Enter open · S save

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

Lingjun Zhang, Yujian Yuan, Changjie Wu, Xinyuan Chang+6

Amap, Alibaba Group · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong+1

Reasoning for VLMs VLMs for Autonomous Driving Autonomous Driving Planning End-to-End Autonomous Driving RL for VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLM) exhibit strong reasoning capabilities, showing promise for end-to-end autonomous driving systems. Chain-of-Thought (CoT), as VLM's widely used reasoning strategy, is facing critical challenges. Existing textual CoT has a large gap between text semantic space and trajectory physical space. Although the recent approach utilizes future image to replace text as CoT process, it lacks clear planning-oriented objective guidance to generate images with accurate scene evolution. To address these, we innovatively propose MindDriver, a progressive multimodal reasoning framework that enables VLM to imitate human-like progressive thinking for autonomous driving. MindDriver presents semantic understanding, semantic-to-physical space imagination, and physical-space trajectory planning. To achieve aligned reasoning processes in MindDriver, we develop a feedback-guided automatic data annotation pipeline to generate aligned multimodal reasoning training data. Furthermore, we develop a progressive reinforcement fine-tuning method to optimize the alignment through progressive high- level reward-based learning. MindDriver demonstrates superior performance in both nuScences open-loop and Bench2Drive closed-loop evaluation. Codes are available at https://github.com/hotdogcheesewhite/MindDriver.

Abstract

Autonomous Driving

Explore similar work

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving

LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers

Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving