Perceives but Doesn't Brake: A Perception-Action Binding Gap in Reasoning Driving VLAs

Matteo Cercola; Katie Z Luo; Simone Formentin; Marco Pavone

cs.ROOct 20, 2026

Perceives but Doesn't Brake: A Perception-Action Binding Gap in Reasoning Driving VLAs

Authors: Matteo Cercola, Katie Z Luo, Simone Formentin, Marco Pavone

Organizations: Politecnico di Milano. · Stanford University. · NVIDIA.

Abstract

Reasoning-augmented vision-language-action (VLA) driving models can perceive and reason about safety-critical hazards yet fail to translate that evidence into safe actions. We characterize this failure as a perception-action binding gap: hazard information is available inside the model but exerts insufficient control over trajectory generation. Using matched simulated counterfactuals and logged driving replay, we show that in-path hazards are actually encoded in the visual representation, but longitudinal plans are dominated by ego-speed history and respond only weakly to hazard proximity or the model's own reasoning trace. Further analysis shows that the trajectory generator is capable of braking but sometimes fails to do so due to the perception-action binding gap. Without retraining or modifying any weights, we inject a hazard-conditioned energy into the frozen flow-matching sampler of Alpamayo-1.5, restoring the missing coupling at generation time. With exact obstacle geometry, guidance cuts footprint intrusion from 58.1% to 14.5% and recovers safe braking whenever sufficient stopping distance remains, eliminating collisions entirely in our braking-relevant closed-loop battery. It also reverses the diagnosed dependence of the plan, from tracking ego speed to responding to obstacle range. These results separate hazard perception from hazard control in driving VLAs, show that unsafe behavior can arise even when the model has both the evidence and the ability to act on it, and propose a simple fix for bridging this binding gap.

Figures & tables

Fig. 1: Where perception–action binding fails, and where we intervene. Binding gap: visual hazard evidence and the model’s reasoning trace have substantially weaker authority over the longitudinal plan than ego history. Intervention: a hazard-conditioned energy gradient is added during flow integration, steering the frozen trajectory generator toward a safe plan without modifying model weights.

	Arm: Other	Contact $\downarrow$	Clear. $\uparrow$	$\Delta$ clear.
	Car’s Motion	(%)	(m)	(m)
All ( $n{=}38$ )	Drives on	0.0	24.31	—
All ( $n{=}38$ )	Stops	42.1 (26.3, 57.9)	0.09	$-23.7$ ( $-29.0$ , $-18.5$ )
Must brake ( $n{=}24$ )	Drives on	0.0	24.31	—
	Stops	54.2 (33.3, 75.0)	$-0.02$	$-23.1$ ( $-28.6$ , $-17.5$ )
	+ guidance	0.0	18.08	${+}17.3$ (+15.0, +19.6)

TABLE I: The dynamic closed-loop hazard: the frozen vehicle is not avoided. Clearance is minimum separation;

\Delta

is paired vs. control (95% CI, scene-clustered bootstrap,

B{=}10^{4}

). The guided row is paired against the car-stops arm, i.e. the effect of the intervention. The Must brake set consists of 24 scenes where avoiding the car requires braking.

Fig. 2: Hazard presence is decodable well before the plan reacts. In matched CARLA pairs, hazard presence is decodable at an area under the curve (AUC) of 0.998, yet every evaluated scene at 8–12 m intrudes. Whiskers are 95% CIs, clustered by site.

Fig. 3: The real-scene plan tracks ego speed, not obstacle range. On held-out real scenes at the planner’s native four-camera input, planned forward reach tracks logged ego speed (left) far better than privileged obstacle range (right).

Fig. 4: Longitudinal action authority is concentrated in ego history. Removing ego history changes longitudinal motion far more than blanking vision or the reasoning trace; the gap mostly disappears for lateral motion. Hatched bars repeat the same ablation on

n{=}120

OOD clips. Whiskers are 95% CIs bootstrapped over clips.

Fig. 5: Exact geometry is sufficient for safe braking, and sourcing it from the model narrows most of the gap. All arms share the same guidance energy and differ only in the signal

s{=}(g,\hat{d},\hat{\ell})

d^{\star}

is the privileged ground-truth range,

\hat{d}

the cache-derived estimate, and

g_{\mathrm{vis}}

its learned gate. (a) Footprint intrusion and (b) near-stop rate on matched CARLA hazard sites, seen and six new sites held out in appearance. Whiskers are 95% CIs, site-clustered bootstraps.

Fig. 6: Guidance restores obstacle authority over longitudinal planning. Planned reach on 273 held-out real scenes vs. (a) ground-truth obstacle range and (b) logged ego speed. Without guidance (gray), reach tracks ego speed but not range (

\rho={+}0.645{}

(+0.552, +0.721) vs.

{+}0.009{}

(

-

0.110, +0.125)); exact-geometry guidance (green) reverses this, tracking range but not speed (

\rho={+}0.678{}

(+0.599, +0.744) vs.

{+}0.177{}

(+0.025, +0.337)).

	readout	MAE (m)	recall	ceiling	intrusion	clear cost
		$\downarrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\downarrow$
Seen ( $n{=}144$ )	pooled, front-wide	2.20	81%	83%	13.4%	4.63 m
	+ front-tele	1.23	74%	74%	16.1%	0.00 m
	attention, front-wide	1.99	76%	76%	16.3%	0.00 m
Unseen ( $n{=}91$ )	pooled, front-wide	3.28	55%	63%	28.6%	0.00 m
	+ front-tele	2.43	66%	68%	18.5%	0.00 m

TABLE II: Input coverage and spatial readout determine the quality of model-derived guidance. Results from the same CARLA sites as Figure 5 . Attention row is averaged over five seeds.

Router	Gating recall $\uparrow$	Nominal stop $\downarrow$
	(%, 95% CI)	(%, 95% CI)
Vision detector	71% (62, 80)	11% (6, 16)
Reasoning trace	55% (45, 65)	15% (10, 21)
Union	87% (80, 93)	18% (12, 25)

TABLE III: Vision and reasoning route on different evidence. On the routing corpus, combining vision and reasoning raises recall from 71% to 87%.

Reasoning for VLA Models

Reasoning for vision-language-action (VLA) models connects visual and language understanding to grounded planning and action.

See all 111 papers

Inference-Time Attention Steering for Vision-Language-Action Driving Models
Aug 17, 2026Darshan Nagendra Prasad, Lars Ullrich, Knut GraichenVision-Language-Action Models VLMs for Autonomous Driving
FAU Erlangen-Nürnberg, Germany
Measuring the Reasoning-Action Coupling in Driving Vision-Language-Action Models through Counterfactuals
Oct 22, 2026Gabriel Manalu, Saad Wazir, Thao Do +4Reasoning for VLA Models VLMs for Autonomous Driving
School of Computing, KAIST, Republic of Korea
ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving
May 27, 2026Mohammadreza Teymoorianfard, Jean-Philippe Monteuuis, Jonathan Petit +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
University of Massachusetts Amherst · Qualcomm

Aug 17, 2026J/K move · Enter open · S save

Inference-Time Attention Steering for Vision-Language-Action Driving Models

Darshan Nagendra Prasad, Lars Ullrich, Knut Graichen

FAU Erlangen-Nürnberg, Germany

Vision-Language-Action Models VLMs for Autonomous Driving

Open paper AskPDF arXiv

Vision-language-action (VLA) driving models couple a reasoning stage with a diffusion-based trajectory decoder, but do not give a direct way to redirect attention toward safety-critical actors at inference time without retraining. We studied a bounded additive pre-softmax attention bias on the visual tokens of detector localized traffic actors on Alpamayo-R1's Qwen3-VL backbone. It is applied as a fail open forward pre-hook with no weight changes. On 50 lane-change scenarios from the Physical AI World Model Synthetic dataset. The trajectory decoder shows a monotonic dose response in the bias magnitude, separate from a paired zero bias control at every tested magnitude. It reaches

\approx 17

,cm mean displacement with lateral shifts up to

\sim 140

\ cm at the clamp. A layer ablation places the action-relevant signal in late layers, where the effect increases with the number of hooked layers (2.0cm for the first 8 layers; 67.6cm for all 36). A per call injection audit explains why the Chain-of-Causation text never changes. The mask based bias never reaches the reasoning pathway in this serving stack, so the invariance is verified exposure, not robustness. Steered trajectories tend to shift toward the attended actor, suggesting the bias governs where the model looks rather than encoding a target behavior.

Abstract

Reasoning for VLA Models

Explore similar work

Inference-Time Attention Steering for Vision-Language-Action Driving Models

Measuring the Reasoning-Action Coupling in Driving Vision-Language-Action Models through Counterfactuals

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving