Mechanistic Interpretability (MI) has made significant progress in explaining the internal computations of Large Language Models (LLMs); however, these advances have yet to demonstrate comparable success in explaining the decision-making processes of LLM-based agents. This paper examines the challenges MI tools face when applied to realistic agentic failures. We focus on one such failure: web agents hallucinating actions that are not available on a website. We apply a series of mechanistic analyzes to Qwen2.5-7B-Instruct as it interacts with live websites. We find that a linear probe trained on the last prompt token can predict hallucinations before generation begins (0.80 AUC) and steering along a probe direction corrects 85% of hallucinations, but disrupts 7% of valid actions. However, these results only provide a partial explanation of why the agent hallucinates. The probe direction is sensitive to prompt formatting, losing much of its steering effect after a minor change to how actions are represented. Additionally, circuit identification can be difficult as the computation is distributed across multiple components and positions, and the information the agent relies on varies across examples. We discuss how these observations challenge the assumptions underlying standard MI methods for circuit discovery and feature identification, and outline future directions for making these methods better suited to explain failures in LLM-based agents.
Figures & tables
Attn Heads
FFN Layers
Fix (↑)
Break (↓)
L22 (All heads)
22, 23, 24
8.9%
2.4%
L22, L27 (All heads)
22, 23, 24
5.9%
27.6%
Top-10 causal attr. (Figure 2(b) )
13, 15, 24
4.4%
3.5%
Top-10 causal attr. (Figure 2(b) )
22, 23, 24
4.5%
4.6%
Top-10 corr. diff. (Figure 2(c) )
22, 23, 24
6.3%
2.5%
Table 1: Intervention results using different sets of attention heads and FFN layers ( α=3 , β=0.5 ).
Figure 2: Component contributions to hallucination. (a) Per-layer FFN amplification effect. (b) Per-head attention noise effect. (c) Per-head max value difference across all heads. Setup details in Appendix D .
Selector start
Task description
Action list
Corrupted P
0.682
0.672
0.325
Residual
0.013 (L6, − 4)
0.012 (L27, 0)
0.360 (L27, 0)
Attention
0.009 (L27, 0)
0.011 (L23, 0)
0.075 (L22, 0)
FFN
0.008 (L18, 0)
0.009 (L18, 0)
0.067 (L21, 0)
Table 2: Activation patching results for P(Search) . Columns are corruption spans; “Corrupted P ” is the mean P(Search) before restoration. Rows show the peak causal recovery ΔP(Search) per component, with peak layer and token offset.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Attn Heads
FFN
α
Fix%
Brk%
Net%
L22 (all)
L22–L24 × 0.5
3
8.90
2.37
6.53
L22 (all)
L22–L23 × 0.5
3
7.95
2.03
5.92
L22 (all)
L0,L5,L22–L24 × 0.5
3
8.33
2.54
5.80
L22 (all)
L22–L23 × 0.5, L20 × 1.5
3
7.01
1.52
5.48
L22 (all)
L22 × 0.5, L20 × 1.5
3
5.87
1.02
4.86
L22 (all)
L22 × 0.5
3
6.25
2.37
3.88
Appendix
Table 3: Head-selective intervention runs sorted by Net Δ % (Fix % - Break %). α : attention head scale. “—” = no intervention on the specific component ( selector_value_start) .
Step
Sel. match (%)
DOM Jac.
Δ (%)
0
–
1.000
+ 38.0
1
28.1
0.521
− 8.5
2
22.2
0.634
− 6.7
3
27.3
0.672
+ 12.1
Appendix
Table 4: Trajectory divergence after removing the chosen action at step 0. Sel. match : % of pairs where original and CF trajectories choose the same action. DOM Jac. : word-level Jaccard similarity between the pages the two trajectories land on. Δ = CF hall. rate − orig. hall. rate.
Position
Layer
α=10
α=20
α=40
α=60
α=80
prompt_end
18
14.6 / 0.8
36.6 / 1.5
41.5 / 10.8
29.3 / 29.5
0.0 / 100.0
prompt_end
26
0.0 / 0.0
2.4 / 2.3
4.9 / 3.1
4.9 / 3.1
12.2 / 2.3
selector_start
17
43.9 / 0.0
75.6 / 3.1
85.4 / 7.0
48.8 / 30.2
0.0 / 96.9
selector_start
18
34.2 / 3.1
56.1 / 2.3
82.9 / 7.0
85.4 / 14.0
2.4 / 91.5
selector_value_start
18
14.6 / 4.6
46.3 / 10.1
46.3 / 19.4
12.2 / 57.4
0.0 / 100.0
selector_value_start
21
9.8 / 3.9
14.6 / 7.0
17.1 / 29.5
29.3 / 33.3
14.6 / 40.3
Appendix
Table 5: Fix and break rates for probe-based steering across (position, layer) configurations. Each cell shows fix / break %. Bold marks the best fix rate for each configuration.
α=10
α=20
α=40
Original format
43.9 / 0.0
75.6 / 3.1
85.4 / 7.0
OOD format
19.4 / 0.0
41.9 / 2.3
58.1 / 7.0
Δ fix
− 24.5
− 33.7
− 27.3
Appendix
Table 6: Fix and break rates for selector_start L17 steering on the original JSON format vs. the OOD numbered-list format. The same steering vector is used in both conditions.
Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can assist with this explanation problem once a circuit has already been identified. We introduce AgenticInterpBench, a benchmark for circuit explanation built from 84 semi-synthetic transformer circuits with 163 component-level annotations. We propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that analyzes each component through an iterative loop of observation, hypothesis generation, and causal validation, eventually producing a component-level explanation and a circuit-level task description. Across four LM backbones, HyVE recovers useful component- and task-level explanations, but no backbone is uniformly best. Our analysis shows that strong backbones usually form observation-grounded hypotheses, while failures more often arise later in the validation loop, through incomplete validation plans, code execution errors, or unresolved hypotheses. A case study on an arithmetic circuit in Llama-3-8B shows that the same formulation can extend beyond semi-synthetic benchmarks to naturally trained models. Overall, LM agents are promising circuit explainers, but reliable validation remains the key obstacle.
Ayan Antik Khan, Harsh Kohli, Yuekun Yao +2
George Mason University · The Ohio State University
Large language model (LLM) web agents are increasingly used for web navigation but remain far from human reliability on realistic, long-horizon tasks. Existing evaluations focus primarily on end-to-end success, offering limited insight into where failures arise. We propose a hierarchical planning framework to analyze web agents across three layers (i.e., high-level planning, low-level execution, and replanning), enabling process-based evaluation of reasoning, grounding, and recovery. Our experiments show that structured Planning Domain Definition Language (PDDL) plans produce more concise and goal-directed strategies than natural language (NL) plans, but low-level execution remains the dominant bottleneck. These results indicate that improving perceptual grounding and adaptive control, not only high-level reasoning, is critical for achieving human-level reliability. This hierarchical perspective provides a principled foundation for diagnosing and advancing LLM web agents.
Mohamed Aghzal, Gregory J. Stein, Ziyu Yao
Department of Computer Science, George Mason University
We propose agentic automata learning to evaluate the extent to which tool-calling LLM agents can uncover hidden environments through interaction, a capability increasingly required in agentic tasks (e.g., reproducing an executable without access to its source code by interacting with it). In our setup, an agent should uncover a hidden deterministic finite automaton (DFA) by interacting with an oracle through (1) membership queries ("Does this string belong to the target language?'') and (2) equivalence queries ("Is this the target DFA?''). Agentic automata learning yields a scalable testbed with controlled task complexity, measurable interaction efficiency, and strong algorithms to compare against from the classic automata-learning literature. Evaluating state-of-the-art LLMs with a multi-turn agent scaffold, we find that while they are able to recover simple DFAs, their performance drops sharply as DFA size increases. Results improve with a more elaborate ReAct-style state-tracking scaffold, yet strong models still struggle with complex instances. Trajectory analyses reveal recurring failures in query planning, evidence integration, and hypothesis construction. These failures occur even though the models mention classic automata learning algorithms in their reasoning, and can implement and execute them when given access to a coding environment. Our results suggest that for state-of-the-art LLMs, identifying a solution to the problem is insufficient, and reliably executing the resulting plan still poses a distinct challenge.
Reef Menaged, Gili Lior, Shauli Ravfogel +2
The Hebrew University of Jerusalem · New York University · Google Research