End-to-end (E2E) autonomous driving models achieve strong planning performance but remain largely opaque, limiting observability and trust. We apply mechanistic interpretability to E2E driving by training Sparse Autoencoders (SAEs) on the internal activations of a simple, single-camera end-to-end path planning model. Unsupervised clustering of SAE latents reveals interpretable structure aligned with route intent, vehicle speed, and braking behavior, recovering route intent with 98.3% accuracy without supervision. Through paired counterfactual image edits, we further show that specific SAE features are visually grounded and systematically respond to scene changes such as traffic light state and stop sign presence. Steering individual SAE latents produces predictable, monotonic changes in planner outputs including lateral displacement, curvature, and braking. Finally, we demonstrate a practical application of these representations: we build a lightweight adapter trained on SAE latents that reduces trajectory selection error by 3.44%, outperforming an equivalent adapter that omits SAE latents, confirming that the sparse representation captures useful information for interpretability. Code and visualizations are available at https://mgagvani.github.io/mechanistic-interpretability-for-e2e-driving/
Figures & tables
Figure 1: Model Architecture. Our Simple Monocular Trajectory Scorer (SMTS) uses only the front camera, past ego state, and route intent as input. Past state and intent initialize a query token, which attends to adapted visual tokens through four transformer blocks. The final query predicts candidate control sequences, rolls them out into future trajectories using a kinematic motion model, and scores each trajectory with a learned proposal scorer. A supervised depth head is used as an auxiliary training output.
Figure 2: Unsupervised SAE feature clusters. Left : Pairwise Pearson correlation matrix of binarized SAE latent activations. Features are ordered by connected components, revealing four major clusters. Right : Cluster-level summaries for the connected components obtained at τ=0.35 , including cluster size, dominant route intent, and standardized driving/model statistics for the scenes that most strongly activate each cluster. The clusters separate left- and right-turn behavior from driving straight, while the two straight clusters further separate low-speed and high-speed scenes.
Source
Var. (%)
Source
Var. (%)
Source
Var. (%)
Image
14.3%
Intent × ego history
16.9%
Image × intent × ego history
4.4%
Intent
26.1%
Image × intent
7.2%
Ego history
24.0%
Image × ego history
6.9%
Table 1: Share of SAE latent variance explained by each input.
Figure 3: Counterfactual visual edits reveal visually grounded SAE features. Each row shows a paired intervention in which the front-facing image is edited while ego history and route intent are held fixed. From left to right, columns show the original image, edited image, largest normalized changes in SAE feature activations, and the selected trajectory before and after the edit. Top : changing green traffic lights to red produces large shifts in a small set of SAE features and changes the selected trajectory from forward motion to stopping. Bottom : inserting a stop sign reduces the selected trajectory displacement.
Figure 4: SAE features have aligned steering effects. Each panel compares counterfactual visual sensitivity with latent steering. Each point is an SAE feature selected among the most responsive features for that edit. The x-axis shows the feature’s normalized signed activation change under the visual edit, while the y-axis shows the standardized change in braking magnitude when the same feature is manually increased through latent steering. Across green-to-red traffic light edits, added stop signs, and added pedestrians, relevant features show a consistent positive alignment between visual response and expected braking behavior.
Figure 5: SAE-based scorer correction improves trajectory selection. The scoring adapter modifies proposal scores using SAE latents while leaving the base planner and generated proposal set fixed. Each example shows a case where score correction changes the selected trajectory to a lower-error proposal. Left: The adapted scorer selects a trajectory that turns further into the side lane rather than continuing straight. Center: The adapted scorer selects a higher-speed left-turn trajectory instead of a slower trajectory that cuts across the lane. Right: The adapted scorer selects a stationary trajectory at a red light instead of accelerating into the intersection from a turn lane.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Data scaling. Selected trajectory ADE@5s on the WOD-E2E validation split as a function of training set size. SMTS shows more consistent gains from additional training data than the compared trajectory scoring baselines and achieves the lowest error overall.
Scene subset
GTRS-Dense
DrivoR
SMTS (ours)
Average
7.688
7.645
7.765
Construction
8.080
8.147
8.343
Intersection
7.948
7.632
7.706
Pedestrian
7.782
7.777
7.977
Cyclist
7.613
7.636
7.569
Multi-lane maneuver
7.409
7.483
7.614
Appendix
Table 2: WOD-E2E test-set Rater Feedback Score by scene subset (higher is better). All models are re-trained on the same 250,000-sample training subset and scored by the private WOD-E2E test set evaluator.
Threshold τ
Number of clusters
Cluster sizes
Clustered features
Top 50 intent purity
0.20
1
238
238
100% , collapsed component
0.35
4
64,36,34,27
161
100% for all clusters
0.40
6
43,29,27,23,3,3
128
96% – 100%
Appendix
Table 3: Sensitivity of block 3 SAE feature clusters to the co-activation correlation threshold. Cluster sizes are reported in descending order. Intent purity is measured among each cluster’s top 50 validation scenes by activation.
Cluster
Features
Top-50 intent
Interpretation
Speed z
Braking z
Score entropy z
C0
64
Straight ( 100% )
Slow straight driving
−1.04
−0.52
+0.03
C1
36
Left ( 100% )
Slow left turn
−1.07
+2.00
−1.75
C2
34
Straight ( 100% )
High-speed straight driving
+0.98
−0.30
+0.48
C3
27
Right ( 100% )
Slow right turn
−1.04
+1.12
−0.94
Appendix
Table 4: Block-3 SAE cluster summaries at the primary threshold τ=0.35 . Intent and standardized scene statistics are computed over each cluster’s top 50 validation scenes by activation.
Figure 7: Highest activating validation scene for each feature cluster. The clusters capture distinct combinations of route intent and vehicle motion.
Edit direction
Count
green → red
193
red → green
95
add stop sign
46
add pedestrian
145
no change / rejected
385
Appendix
Table 5: Counterfactual visual edit dataset summary. The four directions above are those used in the main edit sensitivity analysis. A further 136 edits in other categories were also generated ( add_traffic_light 46 , yellow_to_green 41 , remove_stop_sign 25 , remove_pedestrian 14 , remove_traffic_light 7 , move_stop_sign 3 ), resulting in 615 edited images in total.
Figure 8: Additional counterfactual visual edit examples. Each row shows the original front camera image, the edited image, the top 5 SAE feature activation changes for that edit direction (normalized Δzi ), and the planner’s predicted trajectory before (blue) and after (red) the edit. Stop-inducing edits shorten or brake the predicted path; red → green extends it.
Figure 9: Representative failure modes of the counterfactual editing pipeline. (a) Unrealistic edit: an oddly placed stop sign is added, and the editor warps the scene and deletes parked cars. (b) Irrelevant object edit: a pedestrian is added on the right sidewalk, away from the intended driving path. (c) Visible edit, weak trajectory change: a clear stop sign is added, yet the predicted path shortens only slightly relative to the original. Note that blue/red curves are the selected planner proposals before/after the edit and the dashed green curve is the ground truth future trajectory.
Table 6: Metrics recorded at each intervention level.
Target statistic
Feature
Control score
Effect size
Monotonicity
Consistency
Selectivity
Brake magnitude
137
0.481
+0.715
0.941
1.000
0.715
Curvature
118
0.113
+0.472
0.509
0.641
0.736
Lateral displacement
326
0.330
+0.431
0.947
0.953
0.848
Score margin
228
1.440
+3.028
0.959
0.828
0.599
Proposal spread
250
0.410
+0.651
1.000
1.000
0.629
Appendix
Table 7: Top robust block-3 SAE feature for each planner statistic. Effect sizes are normalized by the validation-set standard deviation of the corresponding statistic.
Figure 10: Mean control score for each block 3 feature cluster and planner statistic. Clusters differ in their secondary controlled statistics.
Statistic
Sign match
Pearson
Spearman
Lateral displacement
59.7%
0.686
0.349
Curvature
40.3%
−0.289
−0.025
Brake magnitude
56.7%
0.434
0.368
Score margin
58.1%
0.233
0.283
Proposal spread
61.5%
0.231
0.352
Appendix
Table 8: Agreement between naturally occurring activation and output associations and positive intervention responses. Pearson and Spearman columns measure the relationship across features between natural association strength and the effect of intervening on SAE features.
Method
Selected mean rank ↓
ADE@5s reduction ↑
Base scorer
16.7
–
Dense query adapter
16.2
2.74%
Random init encoder adapter
16.1
2.86%
SAE latent adapter
15.6
3.44%
Appendix
Table 9: Scorer adapter ablations on WOD-E2E validation. ADE values report relative reduction from the base scorer. The dense-query control replaces SAE latents with the original block-3 activation, while the random-init control replaces the pretrained SAE encoder with an encoder of matched architecture and parameter count trained on the downstream objective.