End-to-end (E2E) autonomous driving models achieve strong planning performance but remain largely opaque, limiting observability and trust. We apply mechanistic interpretability to E2E driving by training Sparse Autoencoders (SAEs) on the internal activations of a simple, single-camera end-to-end path planning model. Unsupervised clustering of SAE latents reveals interpretable structure aligned with route intent, vehicle speed, and braking behavior, recovering route intent with 98.3% accuracy without supervision. Through paired counterfactual image edits, we further show that specific SAE features are visually grounded and systematically respond to scene changes such as traffic light state and stop sign presence. Steering individual SAE latents produces predictable, monotonic changes in planner outputs including lateral displacement, curvature, and braking. Finally, we demonstrate a practical application of these representations: we build a lightweight adapter trained on SAE latents that reduces trajectory selection error by 3.44%, outperforming an equivalent adapter that omits SAE latents, confirming that the sparse representation captures useful information for interpretability. Code and visualizations are available at https://mgagvani.github.io/mechanistic-interpretability-for-e2e-driving/
Figures & tables
Figure 1: Model Architecture. Our Simple Monocular Trajectory Scorer (SMTS) uses only the front camera, past ego state, and route intent as input. Past state and intent initialize a query token, which attends to adapted visual tokens through four transformer blocks. The final query predicts candidate control sequences, rolls them out into future trajectories using a kinematic motion model, and scores each trajectory with a learned proposal scorer. A supervised depth head is used as an auxiliary training output.
Figure 2: Unsupervised SAE feature clusters. Left : Pairwise Pearson correlation matrix of binarized SAE latent activations. Features are ordered by connected components, revealing four major clusters. Right : Cluster-level summaries for the connected components obtained at τ=0.35 , including cluster size, dominant route intent, and standardized driving/model statistics for the scenes that most strongly activate each cluster. The clusters separate left- and right-turn behavior from driving straight, while the two straight clusters further separate low-speed and high-speed scenes.
Source
Var. (%)
Source
Var. (%)
Source
Var. (%)
Image
14.3%
Intent × ego history
16.9%
Image × intent × ego history
4.4%
Intent
26.1%
Image × intent
7.2%
Ego history
24.0%
Image × ego history
6.9%
Table 1: Share of SAE latent variance explained by each input.
Figure 3: Counterfactual visual edits reveal visually grounded SAE features. Each row shows a paired intervention in which the front-facing image is edited while ego history and route intent are held fixed. From left to right, columns show the original image, edited image, largest normalized changes in SAE feature activations, and the selected trajectory before and after the edit. Top : changing green traffic lights to red produces large shifts in a small set of SAE features and changes the selected trajectory from forward motion to stopping. Bottom : inserting a stop sign reduces the selected trajectory displacement.
Figure 4: SAE features have aligned steering effects. Each panel compares counterfactual visual sensitivity with latent steering. Each point is an SAE feature selected among the most responsive features for that edit. The x-axis shows the feature’s normalized signed activation change under the visual edit, while the y-axis shows the standardized change in braking magnitude when the same feature is manually increased through latent steering. Across green-to-red traffic light edits, added stop signs, and added pedestrians, relevant features show a consistent positive alignment between visual response and expected braking behavior.
Figure 5: SAE-based scorer correction improves trajectory selection. The scoring adapter modifies proposal scores using SAE latents while leaving the base planner and generated proposal set fixed. Each example shows a case where score correction changes the selected trajectory to a lower-error proposal. Left: The adapted scorer selects a trajectory that turns further into the side lane rather than continuing straight. Center: The adapted scorer selects a higher-speed left-turn trajectory instead of a slower trajectory that cuts across the lane. Right: The adapted scorer selects a stationary trajectory at a red light instead of accelerating into the intersection from a turn lane.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Data scaling. Selected trajectory ADE@5s on the WOD-E2E validation split as a function of training set size. SMTS shows more consistent gains from additional training data than the compared trajectory scoring baselines and achieves the lowest error overall.
Scene subset
GTRS-Dense
DrivoR
SMTS (ours)
Average
7.688
7.645
7.765
Construction
8.080
8.147
8.343
Intersection
7.948
7.632
7.706
Pedestrian
7.782
7.777
7.977
Cyclist
7.613
7.636
7.569
Multi-lane maneuver
7.409
7.483
7.614
Appendix
Table 2: WOD-E2E test-set Rater Feedback Score by scene subset (higher is better). All models are re-trained on the same 250,000-sample training subset and scored by the private WOD-E2E test set evaluator.
Threshold τ
Number of clusters
Cluster sizes
Clustered features
Top 50 intent purity
0.20
1
238
238
100% , collapsed component
0.35
4
64,36,34,27
161
100% for all clusters
0.40
6
43,29,27,23,3,3
128
96% – 100%
Appendix
Table 3: Sensitivity of block 3 SAE feature clusters to the co-activation correlation threshold. Cluster sizes are reported in descending order. Intent purity is measured among each cluster’s top 50 validation scenes by activation.
Cluster
Features
Top-50 intent
Interpretation
Speed z
Braking z
Score entropy z
C0
64
Straight ( 100% )
Slow straight driving
−1.04
−0.52
+0.03
C1
36
Left ( 100% )
Slow left turn
−1.07
+2.00
−1.75
C2
34
Straight ( 100% )
High-speed straight driving
+0.98
−0.30
+0.48
C3
27
Right ( 100% )
Slow right turn
−1.04
+1.12
−0.94
Appendix
Table 4: Block-3 SAE cluster summaries at the primary threshold τ=0.35 . Intent and standardized scene statistics are computed over each cluster’s top 50 validation scenes by activation.
Figure 7: Highest activating validation scene for each feature cluster. The clusters capture distinct combinations of route intent and vehicle motion.
Edit direction
Count
green → red
193
red → green
95
add stop sign
46
add pedestrian
145
no change / rejected
385
Appendix
Table 5: Counterfactual visual edit dataset summary. The four directions above are those used in the main edit sensitivity analysis. A further 136 edits in other categories were also generated ( add_traffic_light 46 , yellow_to_green 41 , remove_stop_sign 25 , remove_pedestrian 14 , remove_traffic_light 7 , move_stop_sign 3 ), resulting in 615 edited images in total.
Figure 8: Additional counterfactual visual edit examples. Each row shows the original front camera image, the edited image, the top 5 SAE feature activation changes for that edit direction (normalized Δzi ), and the planner’s predicted trajectory before (blue) and after (red) the edit. Stop-inducing edits shorten or brake the predicted path; red → green extends it.
Figure 9: Representative failure modes of the counterfactual editing pipeline. (a) Unrealistic edit: an oddly placed stop sign is added, and the editor warps the scene and deletes parked cars. (b) Irrelevant object edit: a pedestrian is added on the right sidewalk, away from the intended driving path. (c) Visible edit, weak trajectory change: a clear stop sign is added, yet the predicted path shortens only slightly relative to the original. Note that blue/red curves are the selected planner proposals before/after the edit and the dashed green curve is the ground truth future trajectory.
Table 6: Metrics recorded at each intervention level.
Target statistic
Feature
Control score
Effect size
Monotonicity
Consistency
Selectivity
Brake magnitude
137
0.481
+0.715
0.941
1.000
0.715
Curvature
118
0.113
+0.472
0.509
0.641
0.736
Lateral displacement
326
0.330
+0.431
0.947
0.953
0.848
Score margin
228
1.440
+3.028
0.959
0.828
0.599
Proposal spread
250
0.410
+0.651
1.000
1.000
0.629
Appendix
Table 7: Top robust block-3 SAE feature for each planner statistic. Effect sizes are normalized by the validation-set standard deviation of the corresponding statistic.
Figure 10: Mean control score for each block 3 feature cluster and planner statistic. Clusters differ in their secondary controlled statistics.
Statistic
Sign match
Pearson
Spearman
Lateral displacement
59.7%
0.686
0.349
Curvature
40.3%
−0.289
−0.025
Brake magnitude
56.7%
0.434
0.368
Score margin
58.1%
0.233
0.283
Proposal spread
61.5%
0.231
0.352
Appendix
Table 8: Agreement between naturally occurring activation and output associations and positive intervention responses. Pearson and Spearman columns measure the relationship across features between natural association strength and the effect of intervening on SAE features.
Method
Selected mean rank ↓
ADE@5s reduction ↑
Base scorer
16.7
–
Dense query adapter
16.2
2.74%
Random init encoder adapter
16.1
2.86%
SAE latent adapter
15.6
3.44%
Appendix
Table 9: Scorer adapter ablations on WOD-E2E validation. ADE values report relative reduction from the base scorer. The dense-query control replaces SAE latents with the original block-3 activation, while the random-init control replaces the pretrained SAE encoder with an encoder of matched architecture and parameter count trained on the downstream objective.
The increasing adoption of end-to-end learning for autonomous driving introduces increased model complexity and opacity, raising the risk of learning undesired or erroneous behavior. In this work, we integrate unsupervised dictionary learning as a post hoc interpretability module within state-of-the-art driving models to decompose driving behavior into semantically meaningful concepts while demonstrating their causal influence on the model's driving decisions. We propose a stepwise framework for extracting and interpreting meaningful concepts from the end-to-end model and connecting them to the multifaceted model outputs, thereby revealing the underlying decision-making logic for the prediction of future trajectories. Furthermore, targeted interventions at the concept level allow us to manipulate and correct driving decisions, resulting in measurable improvements in overall driving performance. We thus demonstrate how interpretability can effectively be used to reduce model opacity, uncover erroneous behavior, and enable targeted mitigation, ultimately boosting model performance.
End-to-end autonomous driving must balance multimodal maneuver generation against real-time inference constraints. Diffusion planners capture diverse behaviors, but their iterative denoising incurs considerable inference cost in safety-critical deployment, while one-step alternatives still imitate only the single expert demonstration, so neither family certifies both the diversity and the quality of its outputs. To address this, we propose Vault, a framework that couples one-step latent generation with sample-based reinforcement learning guidance. Built on a V-JEPA 2.1 visual representation pretrained on the full NAVSIM training imagery, Vault generates trajectory candidates in a single forward pass by drifting in a VAE latent space. During training, a reward-gated positive pool collects the maximum-scoring trajectories that the model itself discovers under the official evaluator and anchors the drift targets to these certified samples. This closes a reinforcement-learning loop purely through sample-based target construction, without policy gradients, learned reward models, or extra interaction: the generator imitates the frontier of its own certified successes, so quality and multimodality improve together under a single fixed sampling configuration. A learned scorer that predicts the official score and its sub-metrics then selects the executed trajectory, and all training-only modules are discarded at inference. On the NAVSIM benchmarks, Vault achieves state-of-the-art performance at real-time inference cost, reaching 94.6 PDMS on v1 and 91.2 EPDMS on v2.
End-to-end autonomous driving (E2E AD) systems integrate perception, prediction, and planning into a single differentiable architecture. While these models show great promise, their standard training often relies on output-only supervision, which can lead to weak gradients for the hidden layers of increasingly complex models. Recent works have integrated vision-language model (VLM) supervision for latent features to address this, yielding substantial empirical gains, yet leaving the underlying theoretical mechanisms poorly understood. Our investigation into this methodology reveals that the resulting performance gains stem not from VLM reasoning capabilities, as previously assumed, but rather from the latent connections forged between the E2E AD model and ground-truth (GT) data during training. Building on this insight, we propose a probabilistic deep supervision framework that regularizes intermediate latent representations directly from GT data. By treating model latents as reparameterizable distributions, we optimize the architecture via the Evidence Lower Bound (ELBO). Our evaluations conducted on the nuScenes dataset demonstrate that supervising trajectory-related latents with future GT paths consistently improves planning performance. Using identical training data and E2E architectures, our method achieves an 8% reduction in planning L2 error and a 3% decrease in collision rates compared to competitive vectorized baselines, all while incurring negligible computational overhead.
Volodymyr Havrylov, Faris Janjoš, Andreas Look +2
University of Tübingen · Bosch Center for Artificial Intelligence · Coburg University +1