Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection

Jaehwan Jeong; Evelyn Zhu; Jinying Lin; Emmanuel Jaimes; Tuan-Anh Vu; Jungseock Joo; Sangpil Kim; M. Khalid Jawed

Figure 1: Conceptual teaser of our approach to mitigating hallucinations in VLA-based navigation. 1. The Problem: VLA models are highly vulnerable to vision-language reasoning hallucinations, which cause severe deviations from planned trajectories. 2. The Discovery: We reveal that a specific subset of internal attention heads, termed Navigation Heads (

H_{\texttt{nav}}

), is highly sensitive to the robot’s navigation states. This allows them to serve as a built-in anomaly detector with near-zero computational overhead. 3. Our Solution: Building on this insight, we enable real-time, training-free path deviation detection. Upon detecting a failure, the system immediately bypasses the computationally heavy VLA model and triggers a low-level Reinforcement Learning (RL) policy to execute a safe, collision-free rollback.

Figure 2: Overview of the path recovery navigation framework. Deployed via ROS 2, our hierarchical system integrates a high-level VLA model (0.3 Hz) and a low-level RL policy (10 Hz). Using RGB, instructions, and pose data, the VLA detects real-time path deviations (

\mathcal{N}\to\mathcal{A}

) and triggers a rollback to the last verified checkpoint

(x,y,\theta)

. Concurrently, the RL policy (

\pi

) utilizes LiDAR costmaps to output collision-free velocity commands

[v,\omega]

. This design enables the robot to dynamically avoid obstacles and safely return to the normal path via the quickest route during recovery.

Figure 3: Our core approach consists of four consecutive stages: (i) Phase Labeling: Formulating ground-truth labels based on path deviation. (ii) Head Selection: Identifying a subset of Navigation heads

H_{\texttt{nav}}

sensitive to these state transitions. (iii) Anomaly Detection: Monitoring the entropy (

\mathcal{R}_{t}

) of

H_{\texttt{nav}}

to detect real-time failures and preserve safe checkpoints (

\mathcal{C}_{\texttt{safe}}

). (iv) Action Policy: Deploying an RL policy for collision-free navigation and rollbacks triggered by the detected anomalies.

Figure 4: Training-free spatiotemporal grounding framework for navigation. To select robust navigation heads, candidates are first evaluated using the alignment score

I_{\texttt{diag}}(h)

, which comprises even frame energy (

S_{\texttt{uniform}}

), focused instruction attention (

S_{\texttt{peak}}

), diagonal alignment (

S_{\texttt{diag}}

), and smooth transition (

S_{\texttt{shift}}

), and subsequently assessed for their sensitivity to attention changes between normal and anomalous navigation.

Figure 5: The network processes

128\times 128

visual costmaps through a CNN encoder to extract spatial obstacle features, and relative local subgoal states

(x,y)

via an MLP encoder for directional guidance. These features are fused to feed two specialized heads: the Actor head , which outputs Gaussian distributions for velocity control commands

(\upsilon,\omega)

, and the Critic head , which estimates the state-value

V(s)

. This hierarchical design enables the agent to learn reactive obstacle avoidance and robust subgoal navigation through advantage-based reinforcement learning.

Figure 6: Ablation study of navigation heads (

H_{\texttt{nav}}

) and ensemble size (

K

) on the R2R train split. (a) Discriminability of attention heads measured by Cohen’s

d

; higher values indicate greater sensitivity to anomalous behaviors. (b) Episode-level detection performance across

K

. Thresholds are optimized via grid search to maximize Episode Detection Rate (EDR) while constraining False Episode Rate (FER) within 5–10 % (Gap = EDR

-

FER). (c) Step-level classification performance on

\mathcal{N}\to\mathcal{A}

episodes. Notably, an ensemble of just

K=3

heads achieves 43.9 % EDR and 9.6 % FER at the episode level, alongside 92.9 % Precision, 61.6 % Recall, and 74.1 % F1 at the step level.

Figure 7: Hyperparameter grid search for the proposed anomaly detector (

K=3

) on the R2R train split. (a) Both EDR and detection Gap (EDR

-

FER) increase monotonically, saturating around

W=7

. (b) Increasing

P

effectively suppresses FER, albeit at the cost of a linearly increasing mean detection latency. (c) The detector exhibits sharp sensitivity near

\tau=1.0

, where both EDR and FER drop rapidly. Ultimately, driven by the FER

<10\,\%

constraint, we select (

W=10,P=9,\tau=0.95

) that achieves an EDR of 43.9 %, thereby preventing unnecessary interruptions during normal navigation.

Figure 8: Evaluation of the low-level RL obstacle avoidance policy. (a) Quantitative comparison of average performance metrics (SR, CR, and TR) against classical local planning baselines. Our proposed policy significantly outperforms traditional methods, achieving the highest average success rate of 87.3 %. (b) Visualization of the IsaacLab evaluation environment. The testbed comprises a

60\,\text{m}\times 60\,\text{m}

subterrain populated with procedurally generated cylindrical obstacles and wall-shaped boxes to comprehensively assess collision-free navigation.

Figure 9: (a) Hallucination-induced path deviation and recovery in VLN-CE simulation. (b) Real-robot visualization of RL-driven recovery from path deviations during VLA navigation. Additional results are provided in the Supplementary Material.

Figure 10

Configuration	Total Time	Alloc VRAM	Peak VRAM
NaVILA (Baseline)	583.5 ms	17,145.6 MB	18,730.3 MB
NaVILA + Ours	603.3 ms	17,145.6 MB	18,730.3 MB
Difference	+19.8 ms	0.0 MB	0.0 MB

Table 3: Computational overhead analysis on a single NVIDIA RTX 6000 Ada GPU. The integration of our path deviation detection module introduces a marginal latency increase of less than 20 ms per step, with no additional GPU memory required.

Figure 10: RL training diagram. Utilizing IsaacLab [ 41 ] and PhysX, we construct a simulation environment that precisely aligns with real-world dynamics, incorporating the URDF-defined robot geometry and sensor extrinsics while tuning skid-steer friction, damping, and actuator forces alongside the 32-channel LiDAR configurations. To facilitate reward discovery and ensure stable convergence, we employ a curriculum strategy that adaptively scales the navigation challenge by increasing the goal distance from 5 m to 20 m across 20 sub-terrains mixed with cylinders and walls, as illustrated in Fig. 9 of the main paper. With rewards tailored for collision avoidance and goal-reaching, this large-scale training of 1,024 agents on a single NVIDIA RTX 6000 Ada GPU enables comprehensive exploration of the state space and robust policy optimization.

Category	Parameter	Value
Robot & Kinematics	Wheel Radius ( $R$ ) / Wheelbase ( $L$ )	0.165 m / 0.582 m
	Robot Dimensions (Length $\times$ Width)	1.06 m $\times$ 0.90 m
	Max Linear / Angular Velocity	±0.5 m/s / ±0.5 rad/s
	Policy Control Frequency	15 Hz ( $\approx$ 0.067 s/step)
3D LiDAR Sensor	Sensor Mounting Height	0.63 m
	Channels / Max Range	32 / 10.0 m

Table 5: Configuration of the IsaacLab [ 41 ] Simulation Environment

Figure 11: Dynamic 2D Costmap Observation with Ego-Motion Compensation and Temporal Decay. (a) Without proper odometry, uncompensated ego-motion causes severe ghosting artifacts during a clockwise rotation, permanently corrupting the spatial memory. (b) By incorporating strict ego-motion compensation during the same rotation, the local costmap accurately shifts in accordance with the robot’s movement, maintaining a clean and consistent representation. (c) Applying an optimal temporal decay factor (

\gamma=0.9

) allows the system to temporarily retain static obstacles even when they enter the sensor’s blind spot, while gracefully fading the historical trails of dynamic moving objects.

Category	Component	Formulation & Condition
Task	Goal Progress ( $r_{\text{goal}}$ )	$100\cdot(e^{-0.25d_{t}}-e^{-0.25d_{t-1}})$
	Goal Success ( $r_{\text{success}}$ )	$+20.0$ if $d_{t}<$ 0.5 m
	Time Penalty ( $r_{\text{time}}$ )	$-0.05$ per step
Kinematics	Projected Velocity ( $r_{\text{vel}}$ )	$3.0v_{x}\cos(\psi_{\text{err}})(1-0.85\tau_{\text{side}})s_{\text{speed}}(1-\mathbb{I}_{\text{blk}})$
	Heading Alignment ( $r_{\text{head}}$ )	$1.0\cdot(1-\|\psi_{\text{err}}\|/\pi)\cdot\mathbb{I}(\|v_{x}\|>0.1)\cdot(1-\mathbb{I}_{\text{blk}})$
	Yaw Error Reduction ( $r_{\text{yaw}}$ )	$2.0\cdot(\|\psi_{\text{err}}\|_{t-1}-\|\psi_{\text{err}}\|_{t})$

Table 6: Dense Reward Function Configuration. The total reward is the sum of these 12 components.

\mathbb{I}_{\text{blk}}

denotes the path-blocked indicator (

d_{\text{corridor}}<0.8\text{ m}

), and we define a speed scaling factor

s_{\text{speed}}=\text{clamp}(d_{\text{corridor}}/2.0,0,1)

.

Hyperparameter	Value	Hyperparameter	Value
Algorithm	PPO	Learning Rate	$3\times 10^{-4}$
Parallel Environments	1,024	LR Scheduler	KL-Adaptive (0.008)
Rollout Length	96 steps	Discount Factor ( $\gamma$ )	0.96
Learning Epochs	5	GAE Parameter ( $\lambda$ )	0.95
Mini-batches	4	Total Timesteps	100,000
State Preprocessor	Disabled	Checkpoint Interval	5,000 steps

Table 7: PPO Network and Training Hyperparameters (SKRL [ 48 ] )

Configuration	Total Time	Alloc. VRAM	Peak VRAM
Ours (RL Policy)	0.2 ms	26.9 MB	27.9 MB

Table 8: Computational overhead analysis of the RL Action Policy. Evaluated on a single NVIDIA RTX 6000 Ada GPU, the lightweight CNN + MLP architecture (4.89 M parameters) achieves a near-instantaneous inference latency of 0.2 ms per step with minimal VRAM utilization.

Distance	5m			10m			15m			20m
Method	SR $\uparrow$	CR $\downarrow$	TR $\downarrow$	SR $\uparrow$	CR $\downarrow$	TR $\downarrow$	SR $\uparrow$	CR $\downarrow$	TR $\downarrow$	SR $\uparrow$	CR $\downarrow$	TR $\downarrow$
APF [ 26 ]	60.5 %	5.9 %	33.6 %	48.4 %	10.0 %	41.6 %	41.4 %	10.0 %	48.6 %	39.3 %	8.8 %	51.9 %
DWA [ 14 ]	17.6 %	75.8 %	6.6 %	16.2 %	82.8 %	1.0 %	21.1 %	78.1 %	0.8 %	16.2 %	81.8 %	2.0 %
MPPI [ 57 ]	3.7 %	90.0 %	6.3 %	2.7 %	91.6 %	5.7 %	1.6 %	91.6 %	6.8 %	2.0 %	91.4 %	6.6 %
TEB [ 46 ]	65.6 %	24.0 %	10.4 %	51.0 %	40.6 %	8.4 %	41.0 %	50.4 %	8.6 %	31.1 %	59.4 %	9.6 %
Ours	92.2 %	5.1 %	2.7 %	89.8 %	8.6 %	1.6 %	83.6 %	14.3 %	2.1 %	83.9 %	14.5 %	1.6 %

Table 9: Navigation performance by distance. SR: Success Rate, CR: Collision Rate, TR: Timeout Rate.

Figure 12: Real-world navigation performance of the standalone RL policy. Operating at 10 Hz without VLA guidance, the agent safely navigates dynamic environments over tens of meters. (a) Indoors, it successfully avoids static objects and moving pedestrians. (b) In an outdoor orchard, it demonstrates robust collision avoidance against complex, irregular trees.

Figure 13: Overview of the robotic system and simulation environment. First row : sensor calibration of the ZED 2i camera and RS-32 LiDAR. Second row : URDF robot model with sensor extrinsics and training terrains. Third row : Testing real-time ROS 2 integration within Isaac Sim. Fourth row : training visualization of a single agent among 1,024 parallel instances.

Figure 14: ROS 2 System Architecture. The on-device pipeline integrates multimodal data from a skid-steer robot, a 32-channel spinning LiDAR, and a stereo RGB-D camera. For precise robot localization and efficient resource allocation, we employ Fast-LIVO2 [ 59 ] , which leverages LiDAR, RGB, and IMU sensor streams. This is coupled with an Extended Kalman Filter (EKF) that fuses AprilTag [ 43 ] detections to establish and maintain a robust globally fixed coordinate system. The navigation core operates asynchronously, comprising a

\sim

0.3 Hz VLA planner for high-level sub-goal generation and a

\sim

10 Hz RL policy that executes reactive control for obstacle avoidance.

Category	Component	Specification
Hardware	Compute	NVIDIA Jetson AGX Orin Dev. 64 GB
	Mobile Base	AgileX Scout 2.0
	Camera	ZED 2i (Stereo RGB-D)
	LiDAR	RoboSense Helios RS-32
System Software	JetPack	6.2.1 (L4T R36.4.7)
	Kernel	5.15.148-tegra

Table 10: System specifications for physical deployment. This table outlines the comprehensive hardware and software stack configured on the NVIDIA Jetson AGX Orin edge computer. It details the full hierarchical navigation pipeline, encompassing the 8-billion-parameter NaVILA model for high-level reasoning, the attention-based path deviation detection mechanism (parameterized by window size

W

and thresholds), and the lightweight 4.89 M parameter RL action policy for reactive local control.

Figure 15: Physical Robot Setup and Control Center WebUI. ( Left ) Hardware configuration on the AgileX Scout 2.0 mobile base. A custom enclosure rigidly co-locates the Helios-RS 32 LiDAR and ZED 2i stereo camera. The onboard NVIDIA Jetson AGX Orin processes the VLA inference, RL policy, and Fast-LIVO2 odometry. Power distribution is managed via a dedicated DC-DC converter, while a 5G router ensures low-latency remote connectivity. ( Right ) The custom Robot Control Center WebUI was deployed for teleoperation and monitoring.

Parameter	Symbol	Value
Max Linear Velocity	$v_{\texttt{max}}$	0.5 m/s
Max Angular Velocity	$\omega_{\texttt{max}}$	0.5 rad/s
Linear Acceleration Limit	$a_{\texttt{max}}$	1.0 $\text{m/s}^{2}$
Angular Acceleration Limit	$a_{\texttt{max}}^{\omega}$	5.0 $\text{rad/s}^{2}$
Deadzone Threshold	$\epsilon$	0.02

Table 11: Kinematic Action Smoother Configuration. Hyperparameters defining the physical constraints and symmetric acceleration limits for the skid-steer robot.

Odometry	CPU Usage (%)	GPU Usage (%)	Memory (MB)
ZED VIO	+19.4	+42.6	+577.3
Fast-LIVO2	+26.0	+13.3	+367.3

Table 12: State Estimation Resource Overhead. Additional system resources consumed by the localization nodes evaluated on the NVIDIA Jetson AGX Orin 64 GB.

Configuration	Mean (s)	Median (s)	Min / Max (s)	Stdev (s)
NaVILA	3.488	3.489	3.461 / 3.509	0.008
NaVILA + RL	3.488	3.489	3.461 / 3.509	0.008
NaVILA + ZED VIO + RL	4.584	4.588	4.525 / 4.645	0.025
NaVILA + Fast-LIVO2 + RL	3.971	3.968	3.918 / 4.046	0.024
Ours	4.022	4.020	3.910 / 4.120	0.041

Table 13: VLA Inference Time Comparison. Evaluated over 100 runs each on the NVIDIA Jetson AGX Orin. As verified in Sec. B.4 , the lightweight RL policy does not affect the VLA’s inference speed. While integrating state estimation inevitably adds overhead, Fast-LIVO2 maintains a lower latency than ZED VIO by effectively leveraging the CPU, as demonstrated in Table 2. Consequently, Ours (NaVILA + path deviation detection + Fast-LIVO2 + RL) requires only lightweight scalar operations from the attention heads, introducing a negligible additional overhead of

\sim

0.05 s. Ultimately, with only

\sim

0.55 s of total additional computation compared to standalone NaVILA, our system successfully executes primary VLA reasoning alongside trajectory recording, path deviation monitoring, and obstacle avoidance.

Environment	Method	GT Pose Error (m)		GT Yaw Error ( ∘ )		Static Drift (cm)
Environment	Method	Median	Mean	Median	Mean	Median	Mean
Indoor	Fast-LIVO2	0.082	0.331	0.640	4.463	0.080	0.080
Indoor	ZED VIO	0.276	0.413	1.345	5.850	0.025	0.053
Outdoor	Fast-LIVO2	0.375	0.731	6.690	13.469	0.065	0.166
Outdoor	ZED VIO	0.839	4.589	8.560	22.314	0.070	0.124

Table 14: State Estimation Performance Comparison. Fast-LIVO2 vs. ZED VIO Odometry evaluated over 10 indoor and 10 outdoor runs. The position and yaw errors were measured by computing the accumulated drift relative to a fixed AprilTag before and after each run. The best (lowest) error in each metric is highlighted in bold.

Figure 16: Qualitative Trajectory and Yaw Estimation Analysis. Representative evaluation runs comparing Fast-LIVO2 and ZED VIO in (a, b) indoor and (c, d) outdoor environments. The robot starts at the green dot and ends at the red dot. Fast-LIVO2 successfully closes the loop in both scenarios, whereas ZED VIO suffers from severe scale and rotational drift, particularly in the outdoor environment.

Figure 17: Robust Obstacle Avoidance via Low-Level RL Policy. This sequence demonstrates the system’s ability to correct unsafe high-level commands in real-time. Although the VLA model naively outputs a forward movement command ("Move forward 75 cm") despite a pedestrian blocking the path, our RL policy independently detects the obstacle. It immediately overrides the naive command by executing a safe avoidance maneuver (v=0.50 m/s, ω=−0.29 rad/s), allowing the robot to steer around the pedestrian without collision and continue its navigation.

Figure 18: Path Deviation Detection and Autonomous Trajectory Recovery. This sequence demonstrates the system’s rollback mechanism. When a path deviation caused by VLA hallucination is detected during navigation, the system aborts the erroneous trajectory. It then initiates a recovery phase toward the last normal state (e.g., x=5.34, y=5.67, z=47.0°) from the real-time recorded trajectory. During this process, the robot automatically returns to that point via the shortest path, guided by the obstacle avoidance actions of the RL policy, ensuring the mission resumes smoothly.

Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection

Abstract

Figures & tables

Explore similar work

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models