cs.AIAug 25, 2025
SaveUnraveling the cognitive patterns of Large Language Models through module communities
Organizations: Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, USA · IBM Research Yorktown Heights, NY, USA
Abstract
Large Language Models (LLMs) have reshaped our world with significant advancements in science, engineering, and society through applications ranging from scientific discoveries and medical diagnostics to Chatbots. Despite their ubiquity and utility, the underlying mechanisms of LLM remain concealed within billions of parameters and complex structures, making their inner architecture and cognitive processes challenging to comprehend. We address this gap by adopting approaches to understanding emerging cognition in biology and developing a network-based framework that links cognitive skills, LLM architectures, and datasets, ushering in a paradigm shift in foundation model analysis. The skill distribution in the module communities demonstrates that while LLMs do not strictly parallel the focalized specialization observed in specific biological systems, they exhibit unique communities of modules whose emergent skill patterns partially mirror the distributed yet interconnected cognitive organization seen in avian and small mammalian brains. Our numerical results highlight a key divergence from biological systems to LLMs, where skill acquisition benefits substantially from dynamic, cross-regional interactions and neural plasticity. By integrating cognitive science principles with machine learning, our framework provides new insights into LLM interpretability and suggests that effective fine-tuning strategies should leverage distributed learning dynamics rather than rigid modular interventions.
Figures & tables
Figure 1: Connecting cognitive skills, datasets, and LLMs to form a multipartite network. ( a ) The schematic figure illustrates the relationships between cognitive skills, datasets, and LLM weight parameters as a multipartite network, drawing an analogy to the interconnected organization of the human brain. ( b ) The bar plots illustrate how different cognitive functions are categorized by the frequency of their associated skills and how often they appear across multiple-choice question datasets.
Figure 2: Multipartite Network of Skills, Dataset and Modules of Llama2 model. The network depicts modules (squares), datasets (triangles), and skills (circles) as nodes. Edges are weighted by the normalized values derived from the bipartite relationships between skills and datasets and between datasets and modules, reflecting the structural importance and interactions within the multiple types of nodes. The projection network simplifies the multipartite structure by collapsing intermediary nodes(datasets) to focus on the direct interactions between skills and modules using -based pruning strategy. This projection highlights key dependencies and structural patterns within the model, offering insights into which modules are most influential for specific skills.
Figure 3: Community structure comparison between skills and modules networks. (a) The skills projection network with nodes (skills) grouped based on the Louvain community detection algorithm, and colored with the cognitive-function label taken from Table SI1 , allowing direct visual comparison between detected communities and domain ground truth. (b) The Modules projection network, where each node (module) is assigned a community label based on Louvain partitioning applied at multiple resolutions, subsequently consolidated using average-linkage hierarchical clustering. (c) Color legend of cognitive function for node color in skills network in (a) . (d) Schematic figure to represent the frequency of cognitive skills within each community for the Modules network (b) . (e) Adjusted Rand Score (ARS) between the Louvain communities in the Skills network and the cognitive-function labels, plotted against different sparsity ratio for pruning and three base models (Llama, Llama-Chat, and Vicuna). (f) The chi-squared T-test statistically assessed the distinctiveness of skill distributions within each community of Modules networks, with their p-value for those three different models for different sparsity ratios.
Figure 4: Spectral property and influence of modules within each community of Modules Network. ( a-c ) The frequency distributions of eigenvalues reveal the spectral characteristics of the network, where the presence of distinct eigenvalue gaps signifies well-defined community structures.( d-f ) Participation coefficients and Z-scores quantify the roles of modules within and across communities, where higher coefficients highlight bridge modules and Z-scores identify influential or peripheral roles.
Figure 5: Comparison of accuracy and magnitude of weight change across fine-tuned LLMs using different cognitive skilled-based finetuning. ( a-d ) Schematic representation of community-specific fine-tuning to assess the influence of targeted adaptation. ( a ) Community targets community-specific modules aligned with particular cognitive skills, reflecting a focused, skill-driven approach; ( b ) Random uses random module subsets matched in size to the community-specific selections, serving as a baseline to isolate the effects of structural organization versus random variation; ( c ) All represents fine-tuning across all modules, representing a broad, undifferentiated adaptation strategy; ( d ) Without Finetuning depicts models without any fine-tuning, reflecting the unaltered original state of the model. ( e ) Average L2 weight difference for models (Llama, Llama-Chat, Vicuna) across community-aligned datasets using a block-pruning strategy. Each bar represents a different fine-tuning condition. ( f ) Accuracy for the same fine-tuning configurations as ( e ), capturing the model’s performance by fine-tuning on community-based datasets using a block-based pruning strategy.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Cognitive Skills ( ) | Citation |
|---|---|---|
| Cognitive Process (Memory) | sustained attention, selective attention, divided attention, vigilance attention, attention shifting, processing speed, visual processing speed, auditory processing speed, prospective memory, working memory, episodic memory, semantic memory, procedural memory, iconic memory, echoic memory, spatial memory | [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ] |
| Executive Function | planning, organization, goal setting, time management, problem-solving, mental flexibility, strategic thinking, adaptability, impulse control, decision making, emotional regulation, risk assessment, abstract thinking, reasoning, concept formation, cognitive flexibility, creativity | [ 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ] |
| Language Communication | expressive language, receptive language, naming, fluency, comprehension, repetition, reading, writing, pragmatics, discourse ability, expressive language, receptive language, linguistic analysis, narrative skills | [ 49 , 50 , 51 , 52 , 53 , 54 , 55 ] |
| Social Cognition | recognition of social cues, theory of mind, empathy, social judgment, intercultural competence, conflict resolution, self-awareness, relationship management | [ 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 ] |
Table 1: Cognitive Functions with their corresponding cognitive skills. The total number of cognitive skills considered is , with each skill categorized into broader higher-order cognitive domains based on classifications from prior literature.
Figure 6: Heatmap of Skills vs. Datasets ( ) with Hierarchical Clustering. The bipartite matrix represents the number of times skill is required for dataset . Datasets that require similar cognitive skills exhibit strong associations, as indicated by the clustered patterns in the heatmap.
Figure 8: Average Performance of the pruned model after pruning with different sparsity ratios. Average accuracy performance of all datasets with different pruning ratios utilizing different pruning strategies, ( a ), block-based pruning strategy and, ( b ), channel-based pruning strategy.
Figure 9: Average Performance of the pruned model after pruning with different sparsity ratios. ( a ) Importance of Modules quantifies the relationship between each dataset and the weight modules of LLMs ( Llama2 with a pruning ratio). The scatter plot with the line of best fit shows the relationship between the average of all LLM modules( ) and the change in overall model performance before and after pruning. ( b ) The module sparsity ratio distribution of Llama2 with a pruning ratio is shown for two pruning strategies.( c ) The variation of (edge weight between datasets and individual LLM modules) is shown for two pruning strategies.
Figure 10: Evaluating clustering of pruned Llama models using Davies-Bouldin Score and Hotelling’s T-squared test. ( a ) The Davies-Bouldin Score determines the optimal number of K-Means clusters for grouping 174 PCA values derived from module sparsity values, obtained by pruning the Llama2 model using 174 different datasets. ( b ) A scatter plot showing the pruned Llama2 model, grouped by the optimal number of clusters identified in a , alongside randomly pruned models with the 174 datasets. ( c ) P-values from Hotelling’s T-squared test between different clusters, including random pruning, are all significantly small ( ), indicating dissimilarity in information processing between different clusters. ( d ) Hotelling’s T-squared statistics highlight the differences between clusters.
Figure 11: Comparison of community alignment measures between communities in the Skills network and ground-truth cognitive-function labels, across different sparsity levels. Subplots ( a–c ) show results for the block-based pruning strategy, while ( d–f ) display the same for channel-based pruning. Each row visualizes the trends for three base models (Llama, Llama-Chat, and Vicuna), using ( a , d ) Adjusted Normalized Mutual Information (NMI), ( b , e ) Adjusted Rand Index (ARI), and ( c , f ) Jaccard Index as similarity metrics. The x-axis denotes the sparsity ratio applied during pruning, enabling evaluation of how sparsity for pruning the LLMs impacts community alignment within cognitive function labels.
Figure 12: Multi-Layered Network : This diagram highlights the cognitive skills rooted in human cognition, mapped to individual datasets. The modules within LLMs are represented as distinct components, illustrating how different datasets influence these modules. The projection network of modules at the end reflects the localization of cognitive skills across the network.
Figure 13: Heat map clustering of modules network ( ) for the llama model with block-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 14: Heat map clustering of modules network ( ) for the llama-chat model with block-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 15: Heat map clustering of modules network ( ) for the vicuna model with block-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 16: Heat map clustering of modules network ( ) for the llama model with channel-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 17: Heat map clustering of modules network ( ) for the llama-chat model with channel-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 18: Heat map clustering of modules network ( ) for the vicuna model with channel-based pruning, where leaf colors in the dendrograms represent distinct communities formed through hierarchical clustering of the co-assignment matrix, revealing structural patterns among attention modules across layers.
Figure 19: Influence of Modules within each community of Modules Network. ( a-b ) The distribution of edge weight within different communities of modules network for Block and Channel based modules.( c-d ) The scatter plot for different modules over two metrics, participation coefficient and degree z-score metric.
Figure 20: Community-based fine-tuning aligned with cognitive skill relevance The influence of skill distributions within identified module communities is examined by selecting datasets matching the skill profiles of these communities.
Figure 21: Performance of Targeted Finetuning Accuracy and weight difference magnitude of fine-tuned models (Llama, Llama-Chat, Vicuna) across two datasets aligned with each community that were created using block -based pruning strategy.
Figure 22: Performance of Targeted Finetuning Accuracy and weight difference magnitude of fine-tuned models (Llama, Llama-Chat, Vicuna) across two datasets aligned with each community that were created using channel -based pruning strategy.
Figure 23: Visualization of changes in weight modules of the Llama model after fine-tuning, highlighting task associations such as ‘disambiguation_qa’ (Block) and ‘identify_odd_metaphor’ (Channel).
Figure 24: Visualization of changes in weight modules of the Llama-Chat model after fine-tuning, highlighting task associations such as ‘high_school_psychology’ (Block) and ‘electrical_engineering’ (Channel) .
Figure 25: Visualization of changes in weight modules of the Vicuna model after fine-tuning, highlighting task associations such as ‘implicatures’ (Block) and ‘electrical_engineering’ (Channel).