Are Linguistic and Arithmetic Circuits Dissociated in LLMs?

Riku Kisako; Tatsuki Kuribayashi; Ryohei Sasano

cs.CLOct 20, 2026

Are Linguistic and Arithmetic Circuits Dissociated in LLMs?

Authors: Riku Kisako, Tatsuki Kuribayashi, Ryohei Sasano

Organizations: Nagoya University, Japan · MBZUAI, UAE · Tohoku University, Japan

Abstract

Large language models (LLMs) exhibit both knowledge of language, rendered by, for example, grammatical judgment ability, and abilities to perform tasks through language, such as math-word problem solving. Motivated by neurolinguistic studies of their neural dissociation in human brains, in this study, we investigate the overlap of their circuits in large language models (LLMs). Specifically, we examine the overlap and causal effects between the circuits extracted for linguistic and arithmetic tasks, using controlled materials and circuit discovery methods from the mechanistic interpretability field. Our experiments with five LLMs show that circuit similarity is mostly explained by task domains (linguistic vs. arithmetic), rather than by their input and output vocabularies or by the syntactic similarities between natural-language sentences and arithmetic expressions. This finding suggests that LLMs may process linguistic and arithmetic information using largely separate neural circuits, highlighting an ability-based modular organization of circuits within these models.

Figures & tables

Figure 1: Overview of our circuit analysis. We extract a task-specific circuit for each linguistic and arithmetic-related task and compare the extracted circuits to examine whether they are shared or dissociated across the two domains.

Task	Input (Prefix)	Answer	Input	Output	Arith.
Arithmetic calculation. ( Math )	The answer to 2 * 1 + 3 = is	5	Expr.	Arabic numeral	✓
Spelled-out arithmetic calculation ( Math-s )	The answer to two times one plus three equals is	five	Lang.	Number word	✓
Math Word Problem ( GSM-Plus )	Janet’s ducks lay 16 eggs per day… The answer is	18	Lang.	Arabic numeral	✓
Arithmetic expression spelling-out ( Spell-m )	The spelled-out form of 2 * 1 + 3 = is	two times one plus three	Expr.	Number word
Inverse of Spell-m ( Spell-mi )	The numerical formula for two times one plus three is	2 * 1 + 3	Lang.	Arabic numeral
Arithmetic expression parsing ( Math-Parse )	The first term to calculate in 6 + 2 * 5 is	2 * 5	Expr.	Arabic numeral	( ✓ )

Table 1: Task examples in our experiments. Horizontal blocks indicate four high-level categories based on task domain and required competence. Expr. and Lang. denote symbolic expressions and natural language, respectively. For generation tasks, we evaluate only the first generated token; subsequent tokens are shown in gray. Arithmetic checkmark denotes whether the task involves arithmetic calculation, and Math-Parse has a gray one, as it does not involve calculation but should be a necessary sub-task for arithmetic.

Model	Math	Math-s	GSM-Plus	Spell-m	Spell-mi	Math-Parse	Ling-Parse	Ling-SV	Ling-GA	Ling-HJ
Llama-3.1-8B	96.0	85.2	11.4	100.0	100.0	99.8	94.8	99.8	92.6	98.8
Qwen3-8B	90.8	77.2	36.8	100.0	100.0	100.0	90.0	100.0	86.2	97.6
Qwen3-1.7B	82.2	45.4	23.0	99.6	100.0	100.0	84.0	43.2	79.4	29.6
Qwen3-4B	92.0	60.0	36.8	99.8	100.0	100.0	93.0	86.0	79.6	78.6
Gemma-2-2B	66.6	41.2	24.2	100.0	100.0	61.4	75.2	99.6	85.2	98.0

Table 2: Model accuracy on each task. Values are percentages. We report exact top-1 accuracy, except in cases where only the available answer-vs.-corrupted pairwise rows are used in place of exact accuracy.

Panel of Figure 2 — Figure 2: Left: IoU (structural overlap). Right: cross-task Faithfulness (functional overlap). For Faithfulness, rows denote the tasks from which circuits are extracted, and columns denote the target tasks solved with those circuits. Scores are normalized by the corresponding diagonal self-faithfulness value.

Figure 3: Hierarchical clustering of task circuits based on cross-task Faithfulness transfer profiles under the top-

2.5\%

circuit definition. Top: Qwen3-8B. Bottom: Llama-3.1-8B.

$K$	Mean ARI	Median ARI	Min	Max
2	0.628	0.597	0.294	1.000
3	0.580	0.570	0.247	1.000
4	0.452	0.400	0.203	0.743
5	0.589	0.371	0.371	1.000
6	0.550	0.438	0.325	1.000

Table 3: Adjusted Rand Index (ARI) across all ten model pairs for partitions of the top-

2.5\%

faithfulness dendrograms into

K

clusters.

Panel of Figure 4 — Figure 4: Results for Qwen3-8B under the top- $5.0\%$ circuit definition.

Panel of Figure 5 — Figure 5: Results for Qwen3-8B under the adaptive circuit size with the self-faithfulness threshold of $0.85$ .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Input	Expected Output	Loss
	Clean	Clean
	Corrupted	Corrupted
Arithmetic calculation ( Math )	The answer to 2 * 12 + 3 = is The answer to 13 - 4 + 9 = is	27 18	logit_diff
Spelled-out arithmetic calculation ( Math-s )	The answer to two times twelve plus three equals is The answer to thirteen minus four plus nine equals is	twenty-seven eighteen	logit_diff
Math Word Problem ( GSM-Plus )	Q: Janet’s ducks lay 16 eggs per day... Q: Janet’s ducks lay 10 eggs per day...	18 27	logit_diff
Arithmetic expression spelling-out ( Spell-m )	The spelled-out form of 2 * 12 + 3 is The spelled-out form of 13 - 4 + 9 is	two times twelve plus three thirteen minus four plus nine	logit_diff

Appendix

Table 4: Example instances for each task in our controlled suite. For each task, we show a clean/corrupted prefix pair and the corresponding expected output, along with the loss used for evaluation (logit_diff or prob_diff).

Model	HuggingFace	License
Llama-3.1-8B	https://huggingface.co/meta-llama/Llama-3.1-8B	Llama 3.1 Community License
Qwen3-8B	https://huggingface.co/Qwen/Qwen3-8B	Apache license 2.0
Qwen3-1.7B	https://huggingface.co/Qwen/Qwen3-1.7B	Apache license 2.0
Qwen3-4B	https://huggingface.co/Qwen/Qwen3-4B	Apache license 2.0
Gemma-2-2B	https://huggingface.co/google/gemma-2-2b	Gemma Terms of Use License

Appendix

Table 5: License of the models

Panel of Figure 6 — Figure 6: Left: IoU (structural overlap). Right: cross-task Faithfulness (functional overlap). For Faithfulness, rows denote the tasks from which circuits are extracted, and columns denote the target tasks solved with those circuits. Scores are normalized by the corresponding diagonal self-faithfulness value.

Panel of Figure 7 — Figure 7: Hierarchical clustering of task circuits based on cross-task Faithfulness transfer profiles. Top: Qwen3-1.7B. Middle: Qwen3-4B. Bottom: Gemma-2-2B.

Circuit	MATH	MATH-S	GSM-PLUS	SPELL-M	SPELL-MI	MATH-PARSE	LING-PARSE	LING-SV	LING-GA	LING-HJ
MATH	0.58	0.48	0.60	0.46	0.28	0.43	0.14	0.25	-0.01	0.05
MATH-S	0.52	0.51	0.51	0.42	0.25	0.45	0.30	0.30	0.02	0.06
GSM-PLUS	0.61	0.49	0.92	0.88	0.71	0.43	0.50	0.28	0.05	0.08
SPELL-M	0.41	0.42	0.69	0.98	0.91	0.44	0.64	0.39	0.04	0.21
SPELL-MI	0.42	0.42	0.71	0.97	0.94	0.44	0.64	0.27	0.02	0.18
MATH-PARSE	0.29	0.33	0.04	0.05	0.03	0.51	0.26	0.07	0.02	0.05

Appendix

Table 6: Llama-3.1-8B: Cross-task Faithfulness (normalized by clean/corrupted baselines; rows: circuits, columns: tasks; Top 2.5%)

Circuit	MATH	MATH-S	GSM-PLUS	SPELL-M	SPELL-MI	MATH-PARSE	LING-PARSE	LING-SV	LING-GA	LING-HJ
MATH	0.61	0.49	0.86	0.12	0.13	0.44	0.25	0.03	0.00	0.09
MATH-S	0.58	0.56	0.76	0.24	0.23	0.44	0.36	0.05	0.02	0.11
GSM-PLUS	0.56	0.46	1.03	0.41	0.55	0.43	0.17	0.06	0.05	0.16
SPELL-M	0.41	0.34	0.80	0.95	0.88	0.44	0.22	0.08	0.03	0.17
SPELL-MI	0.48	0.40	0.84	0.80	0.93	0.43	0.33	-0.00	0.02	0.12
MATH-PARSE	0.12	0.12	0.10	0.03	0.04	0.46	0.04	-0.05	-0.00	0.05

Appendix

Table 7: Qwen3-1.7B: Cross-task Faithfulness (normalized by clean/corrupted baselines; rows: circuits, columns: tasks; Top 2.5%)

Circuit	MATH	MATH-S	GSM-PLUS	SPELL-M	SPELL-MI	MATH-PARSE	LING-PARSE	LING-SV	LING-GA	LING-HJ
MATH	0.61	0.58	0.65	0.24	0.16	0.38	0.02	-0.02	0.01	0.13
MATH-S	0.61	0.60	0.71	0.39	0.35	0.37	0.11	0.01	0.02	0.16
GSM-PLUS	0.54	0.52	0.95	0.66	0.71	0.36	0.18	-0.02	-0.01	0.22
SPELL-M	0.52	0.49	0.57	0.98	0.94	0.37	0.31	0.02	0.05	0.19
SPELL-MI	0.51	0.48	0.57	0.92	0.94	0.43	0.50	0.09	0.05	0.19
MATH-PARSE	0.14	0.18	-0.14	0.02	0.02	0.54	0.16	-0.10	0.00	0.04

Appendix

Table 8: Qwen3-4B: Cross-task Faithfulness (normalized by clean/corrupted baselines; rows: circuits, columns: tasks; Top 2.5%)

Circuit	MATH	MATH-S	GSM-PLUS	SPELL-M	SPELL-MI	MATH-PARSE	LING-PARSE	LING-SV	LING-GA	LING-HJ
MATH	0.60	0.53	0.80	0.40	0.40	0.36	0.04	-0.01	0.01	0.07
MATH-S	0.61	0.57	0.77	0.57	0.59	0.30	0.05	-0.01	0.03	0.07
GSM-PLUS	0.60	0.54	0.99	0.77	0.76	0.36	0.15	0.00	0.04	0.15
SPELL-M	0.48	0.42	0.78	0.98	0.94	0.40	0.45	0.10	0.03	0.12
SPELL-MI	0.54	0.48	0.82	0.97	0.99	0.47	0.28	0.03	0.05	0.15
MATH-PARSE	0.24	0.24	0.14	0.05	0.06	0.57	0.14	-0.06	0.00	0.06

Appendix

Table 9: Qwen3-8B: Cross-task Faithfulness (normalized by clean/corrupted baselines; rows: circuits, columns: tasks; Top 2.5%)

Circuit	MATH	MATH-S	GSM-PLUS	SPELL-M	SPELL-MI	MATH-PARSE	LING-PARSE	LING-SV	LING-GA	LING-HJ
MATH	0.41	0.36	0.29	0.21	0.23	0.17	0.23	-0.14	0.00	0.01
MATH-S	0.26	0.31	0.04	0.22	0.28	0.17	0.23	-0.16	-0.00	-0.00
GSM-PLUS	0.31	0.29	0.62	0.34	0.36	0.22	0.34	-0.13	-0.00	0.02
SPELL-M	0.25	0.22	0.26	0.92	0.97	0.17	0.82	-0.05	0.01	0.06
SPELL-MI	0.23	0.21	0.25	0.85	0.97	0.19	1.14	0.12	0.03	0.05
MATH-PARSE	0.03	0.05	-0.14	0.01	0.00	0.24	0.05	-0.21	0.01	0.01

Appendix

Table 10: Gemma-2-2B: Cross-task Faithfulness (normalized by clean/corrupted baselines; rows: circuits, columns: tasks; Top 2.5%)

Figure 8: Comparison of first-token (left) and sequence-level (right) cross-task faithfulness for Qwen3-8B under the top- $2.5\%$ circuit definition. Rows denote the ten source-circuit tasks, and columns denote the four target tasks reevaluated at the sequence level. Both panels use the same circuits extracted and selected with first-token EAP-IG scores and share a color scale. Scores are normalized by clean/corrupted baselines and then by the corresponding target’s self-circuit faithfulness within each evaluation setting.

Figure 9: Cross-model agreement of hierarchical clustering based on source-circuit faithfulness profiles under the top- $2.5\%$ circuit definition. (a) Mean Adjusted Rand Index (ARI) over all ten model pairs after cutting each dendrogram into $K$ clusters; shading spans the pairwise minimum and maximum. (b) Pearson correlations between cophenetic-distance vectors, comparing complete dendrograms without selecting a cut level.

Circuits in Language Models

Circuits in language models are interacting internal components that implement behaviors and can be traced with causal interventions.

See all 122 papers

Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
Apr 17, 2026Tanja Baeumel, Josef van Genabith, Simon OstermannLLM Interpretability Mechanistic Interpretability
German Research Center for Artificial Intelligence (DFKI) · Saarland University · Center for European Research in Trusted AI (CERTAIN)
Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning
Sep 3, 2026Andrea Gregor de Varda, Sana Pandey, Pengrui Han +2Language Model Generalization LLM Mathematical Reasoning
Massachusetts Institute of Technology
Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs
Jul 18, 2026Sharath Naganna, Tanvir Ahmed Sijan, Uddipta KalitaLLM Interpretability Circuits in Language Models
Indian Institute of Technology Bombay, Mumbai, India · Jahangirnagar University, Dhaka, Bangladesh · National Institute of Technology, Silchar, India

Apr 17, 2026J/K move · Enter open · S save

Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms

Tanja Baeumel, Josef van Genabith, Simon Ostermann

German Research Center for Artificial Intelligence (DFKI) · Saarland University · Center for European Research in Trusted AI (CERTAIN)

LLM Interpretability Mechanistic Interpretability LLM Mathematical Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling reasoning-intensive tasks remain underexplored. To advance the understanding of model-internal processing mechanisms, we present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution. Using early decoding, we trace how next-token predictions are constructed across layers. Our experiments reveal that while the models recognize arithmetic tasks early, correct result generation occurs only in the final layers. Notably, models proficient in arithmetic exhibit a clear division of labor between attention and MLP modules, where attention propagates input information and MLP modules aggregate it. This division is absent in less proficient models. Furthermore, successful models appear to process more challenging arithmetic tasks functionally, suggesting reasoning capabilities beyond factual recall.

Abstract

Circuits in Language Models

Explore similar work

Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms

Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning

Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs