Simplifying Outcomes of Language Model Component Analyses with ELIA

Aaron Louis Eidt; Nils Feldhus

cs.CLFeb 20, 2026

Simplifying Outcomes of Language Model Component Analyses with ELIA

Authors: Aaron Louis Eidt, Nils Feldhus

Organizations: Technische Universität Berlin · Fraunhofer Heinrich Hertz Institute · BIFOLD – Berlin Institute for the Foundations of Learning and Data

Abstract

While mechanistic interpretability has developed powerful tools to analyze the internal workings of Large Language Models (LLMs), their complexity has created an accessibility gap, limiting their use to specialists. We address this challenge by designing, building, and evaluating ELIA (Explainable Language Interpretability Analysis), an interactive web application that simplifies the outcomes of various language model component analyses for a broader audience. The system integrates three key techniques -- Attribution Analysis, Function Vector Analysis, and Circuit Tracing -- and introduces a novel methodology: using a vision-language model to automatically generate natural language explanations (NLEs) for the complex visualizations produced by these methods. The effectiveness of this approach was empirically validated through a mixed-methods user study, which revealed a clear preference for interactive, explorable interfaces over simpler, static visualizations. A key finding was that the AI-powered explanations helped bridge the knowledge gap for non-experts; a statistical analysis showed no significant correlation between a user's prior LLM experience and their comprehension scores, suggesting that the system reduced barriers to comprehension across experience levels. We conclude that an AI system can indeed simplify complex model analyses, but its true power is unlocked when paired with thoughtful, user-centered design that prioritizes interactivity, specificity, and narrative guidance.

Figures & tables

Figure 1: ELIA system overview, including three analysis methods (Attribution Analysis, Function Vectors, and Circuit Tracer) and the explanation generation workflow using VLMs to transform complex interpretability analyses into accessible NLEs. The system is evaluated using a Faithfulness Checker and a user study.

Figure 2: The interactive Attribution Heatmap using Integrated Gradients with an AI-generated natural language explanation. The heatmap visualizes the influence of input tokens on the generated output, and the explanation interprets these results in an accessible narrative.

Figure 3: Function Vector and Circuit Trace Analysis visualizations. The 3D PCA plot (left) places the user’s prompt in a semantic functional space, while the Circuit Graph (right) traces the flow of information through interpretable features across layers. Both are accompanied by AI-generated explanations.

Component	Feature	Verified	Faith. (%)
Attribution	Saliency	59/68	86.8%
	Int. Gradients	56/61	91.8%
	Occlusion	66/75	88.0%
Func. Vectors	Placement	23/24	95.8%
	Func. Type	47/47	100.0%
	Categories	44/48	91.7%

Table 1: Faithfulness verification results of each explanation type across all three analysis pages. All explanations are generated by Qwen2.5-VL-72B and verified against ground-truth data from the underlying analyses.

Figure 4: Impact of intervention on model output probability ( $|\Delta p|$ ). We compare the effect of ablating top- $k$ targeted features and traced circuits against random baselines (ablating random features or edges).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Attribution Analysis: Faithfulness verification of the AI-generated explanation.

Figure 7: Influence Tracer: Retrieves and displays training documents similar to the prompt.

Function Type	Category	Example Prompts
Abstractive Tasks	country_capital	The capital of France is
Abstractive Tasks		What city serves as the capital of Japan?
	translation_german	Translating ’hello’ into German gives
		What would a German speaker say for ’world’?
	next_item	After ’Monday’ comes
		The number following 5 is

Appendix

Table 2: Overview of the Function Vector dataset structure. The dataset contains 6 function types with 120 total categories. Each category includes 5 example prompts. This table shows representative examples from each function type.

Figure 8: Function Vector Analysis: Faithfulness verification of the AI-generated PCA explanation.

Figure 9: Cross-Layer Transcoder training dynamics showing Total Loss, Reconstruction Loss, and Sparsity Loss ( $L_{1}$ ) over 1500 training steps. The CLT is trained on the Dolma dataset with $L_{1}$ sparsity regularization, gradient clipping, and cosine annealing learning rate scheduling.

Figure 10: Circuit Trace Analysis: Local path ablations. This view shows the effect of ablating specific paths within the top feature graph shown in the main circuit trace visualization.

Figure 11: Circuit Trace Analysis: Faithfulness verification of the circuit explanation.

Figure 12: Circuit Trace Analysis: Subnetwork Explorer. This interactive tool allows users to isolate and visualize the specific computational pathways connected to a chosen feature, revealing its upstream influences and downstream effects.

Figure 13: Circuit Trace Analysis: Feature Explorer. Users can inspect individual features in detail, viewing their top activating tokens, sparsity statistics, and AI-generated functional interpretations.

Natural Language Explanation Generation

See all 13 papers

Interpretative Interfaces: Designing for AI-Mediated Reading Practices and the Knowledge Commons
Mar 16, 2026Gabrielle BenabdallahLLM Interpretability Interpretable ML
University of Washington Seattle, Washington, USA
Pitfalls in Evaluating Interpretability Agents
Mar 20, 2026Tal Haklay, Nikhil Prakash, Sana Pandey +5Mechanistic Interpretability LLM Agent Evaluation
Technion—IIT · Northeastern University · MIT +2
BoolXLLM: LLM-Assisted Explainability for Boolean Models
May 12, 2026Du Cheng, Serdar Kadioglu, Xin WangExplainable Artificial Intelligence Boolean Function Learning
AI Center of Excellence, Fidelity Investments, MA, USA · Department of Computer Science, Brown University, RI, USA

Mar 16, 2026J/K move · Enter open · S save

Interpretative Interfaces: Designing for AI-Mediated Reading Practices and the Knowledge Commons

Gabrielle Benabdallah

University of Washington Seattle, Washington, USA

LLM Interpretability Interpretable ML Explainable Artificial Intelligence AI-Assisted Scientific Research Mechanistic Interpretability

Open paper AskPDF arXiv

Explainable AI (XAI) interfaces seek to make large language models more transparent, yet explanation alone does not produce understanding. Explaining a system's behavior is not the same as being able to engage with it, to probe and interpret its operations through direct manipulation. This distinction matters for scientific disciplines in particular: scientists who increasingly rely on LLMs for reading, citing, and producing literature reviews have little means of directly engaging with how these models process and transform the texts they generate. In this ongoing design research project, I argue for a shift from explainability to interpretative engagement. This shift moves away from accounts of system behavior to instead enable users to manipulate a model's intermediate representations. Drawing on textual scholarship, computational poetics, and the history of reading and writing technologies, including practices such as marginalia, glosses, indices, and annotation systems, I propose interpretative interfaces as interactive environments in which non-expert users can intervene in the representational space of a language model. More specifically, such interfaces will allow users to select a token and follow its trajectory through the model's intermediate layers. This way, they can observe how its semantic position shifts as context is processed, and possibly annotate the transformations they find useful or meaningful. The same way readers can create their own maps within a book through annotations and bookmarks, interpretative interfaces will allow users to inscribe their reading of a model's internal representations. The goal of this project is to reframe AI interpretability as an interaction design project rather than a purely technical one, and to open a path toward AI-mediated reading that supports interpretative engagement and critical stewardship of scientific knowledge.

Abstract

Natural Language Explanation Generation

Explore similar work

Interpretative Interfaces: Designing for AI-Mediated Reading Practices and the Knowledge Commons

Pitfalls in Evaluating Interpretability Agents

BoolXLLM: LLM-Assisted Explainability for Boolean Models