Understanding the internal functional organization of Large Language Models (LLMs) is crucial for improving their trustworthiness and performance. However, how LLMs organize different functions into modules remains highly unexplored. To bridge this gap, we formulate a functional module discovery problem and propose an Unsupervised LLM Cross-layer MOdule Discovery (ULCMOD) framework that simultaneously disentangles the large set of neurons in the entire LLM into modules while discovering the topics of input samples related to these modules. Our framework introduces a novel objective function and an efficient Iterative Decoupling (IterD) algorithm. Extensive experiments show that our method discovers high-quality, disentangled modules that capture more meaningful semantic information and achieve superior performance in various downstream tasks. Moreover, our qualitative analysis reveals that the discovered modules show semantic coherence, correspond to interpretable specializations, and a clear spatial and hierarchical organization within the LLM. Our work provides a novel tool for interpreting the functional modules of LLMs, filling a critical blank in LLM's interpretability research.
Figures & tables
Figure 1: Illustrations of brain modules vs. LLM modules.
Figure 2: Illustration of function modules. Each red block represents a function module, associated with a set of neurons and a set of representative samples employing it.
Figure 3: Overview of the IterD optimization preocess.
LLM
K
K-Means
Mini-Batch K-Means
Agglomerative
Spectral
Spectral Co-cluster
IterD ( Ours )
L(F)
ξ(F)
B(F)
L(F)
ξ(F)
B(F)
L(F)
ξ(F)
B(F)
L(F)
ξ(F)
B(F)
L(F)
ξ(F)
B(F)
L(F)
ξ(F)
B(F)
Qwen2.5 -1.5B -Instruct
5
19.2
0.409
46.9
23.4
0.406
57.6
10.1
0.311
32.4
9.6
0.335
28.7
24.0
0.380
63.2
31.4
0.465
67.5
10
5.7
0.644
8.8
4.2
0.630
6.7
5.2
0.599
8.7
2.8
0.566
5.0
7.9
0.522
15.1
12.0
0.711
16.9
15
0.9
0.829
1.1
1.9
0.752
2.6
1.9
0.655
2.9
0.7
0.750
0.9
3.8
0.583
6.4
6.6
0.892
7.5
20
0.4
0.943
0.4
0.5
0.855
0.6
1.0
0.731
1.3
0.3
0.914
0.3
2.0
0.636
3.1
4.4
1.048
4.2
Qwen2.5 -3B -Instruct
5
28.0
0.466
60.1
26.9
0.490
54.9
16.1
0.439
36.6
0.0
0.001
7.7
34.6
0.404
85.6
56.9
0.533
106.8
Table 1: Comparison of Clustering Metrics for different methods, LLMs, and number of clusters ( K ). Values for L(F) and B(F) are scaled by ×106 .
Figure 4: Average activation heatmap for discovered modules in Qwen2.5-7B-Instruct ( K=10 ). Each cell (i,j) shows the average activation of neuron set Uj on sample set Si .
LLM
Cls.
K
K-Means
Mini-Batch K-Means
Agglomerative
Spectral
Spectral Co-Clustering
IterD ( Ours )
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Qwen2.5 -1.5B -Instruct
LR
5
0.4800
0.4758
0.5400
0.5404
0.5412
0.5409
0.5379
0.5300
0.4650
0.4602
0.5436
0.5428
10
0.7186
0.7173
0.6929
0.6927
0.7057
0.7039
0.7150
0.7150
0.6793
0.6764
0.7257
0.7259
15
0.7664
0.7662
0.7729
0.7724
0.7750
0.7750
0.7486
0.7474
0.7107
0.7089
0.8157
0.8158
20
0.7907
0.7908
0.7843
0.7839
0.7850
0.7849
0.7864
0.7862
0.7643
0.7637
0.8200
0.8200
SVC
5
0.5414
0.5417
0.5800
0.5829
0.5921
0.5933
0.5764
0.5695
0.5343
0.5339
0.5964
0.5987
Table 2: Comparison of Test Accuracy and Macro-F1 for different methods, LLMs, classifiers, and number of clusters ( K ). Features are derived from average function module activations.
Figure 5: Visualization of sample category similarity in Qwen2.5-7B-Instruct ( K=10 ). The value in cell (i,j) represents the average cosine similarity between the feature vectors ( xs ) of samples from category i and category j .
Figure 6: The studies of how neuron count is distributed in different layers inside each function module for Qwen2.5-3B-Instruct ( K=5 ).
Figure 7: Visualization of discovered function modules.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Layer-wise distribution of function modules in Qwen2.5-3B-Instruct ( K=10 ). Each subfigure shows the neuron counts in each layer for the corresponding function module.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks, from text summarization to question answering. Despite these capabilities, their black-box nature obscures internal decision-making processes. Mechanistic interpretability (MI) aims to address this by reverse-engineering neural networks into human-understandable algorithms. Current MI approaches for LLMs typically follow a two-stage paradigm: first identifying important components (circuit discovery), where components are typically individual nodes such as an attention head or feedforward neuron, and second determining the role they play in a certain task (functional interpretation). However, this sequential approach overlooks a fundamental insight: a component's importance and its functional role are inherently codependent. Unifying these stages presents two key challenges: (1) functional roles are often tied to specific nodes or components, limiting generalization, and (2) their identification relies on subjective interpretation rather than quantifiable metrics. To address these challenges, we propose S^3martCirc (Self-supervised Smart Circuit Discovery), a unified framework that simultaneously discovers circuits and interprets functionality. S^3martCirc abstracts node behavior into two general computational roles that generalize across tasks and defines a quantitative metric for assigning them, enabling importance and functional role to be discovered jointly rather than in sequence. Extensive experiments show that our framework outperforms existing methods in circuit discovery.
While Large Language Models (LLMs) have achieved strong performance across many NLP tasks, their opaque internal mechanisms hinder trustworthiness and safe deployment. Existing surveys in explainable AI largely focus on post-hoc explanation methods that interpret trained models through external approximations. In contrast, intrinsic interpretability, which builds transparency directly into model architectures and computations, has recently emerged as a promising alternative. This paper presents a systematic review of the recent advances in intrinsic interpretability for LLMs, categorizing existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction. We further discuss open challenges and outline future research directions in this emerging field. The paper list is available at: https://github.com/PKU-PILLAR-Group/Survey-Intrinsic-Interpretability-of-LLMs.
Yutong Gao, Qinglin Meng, Yuan Zhou +1
MOE Key Lab of Computational Linguistics, Peking University · Nanjing University of Science and Technology · Purdue University +1
Although large language models (LLMs) encode language in highly distributed spaces, they remain vulnerable to focal disruptions---a phenomenon strikingly similar to how localized brain damage causes clinical aphasia in humans. To investigate this theoretical tension, we ask whether LLMs possess a functional structure analogous to the human language system, by introducing a novel mechanistic interpretability framework inspired by aphasia. Through two complementary approaches, classifier-guided lesioning and continuous activation steering, we systematically induce aphasia-like breakdowns in LLMs. We find that language functions can be successfully isolated to discrete structural circuits and continuous representational vectors, yet these functions remain densely entangled with significant circuit overlap and vector similarities across distinct aphasic profiles. Furthermore, while steering generalizes robustly across downstream tasks, structural lesioning remains asymmetrically constrained. By bridging neuropsychology and mechanistic interpretability, we demonstrate that LLMs, functionally analogous to biological brains, rely on a densely entangled architecture that can be systematically mapped through clinically-informed interventions.
Muxuan He, Wang Bill Zhu, Yue Chen +5
University of Southern California · University of California, Berkeley · Independent Researcher +1