Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
Organizations: University of Wisconsin–Madison
Abstract
Large language models (LLMs) often achieve strong benchmark accuracy yet remain brittle under small distribution shifts. While recent mechanistic studies reveal the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill acquisition and the role of data distributions remain elusive. In this study, we train transformers on synthetic arithmetic tasks with black-box model-agnostic metrics for analyzing non-human skill compositions. We discover that transformers often acquire skills for arithmetic in reverse order or in parallel instead of human-like sequential rules--a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. As a consequence, this non-human acquisition creates competition between partially learned skills, producing characteristic mixing errors and weaker robustness under controlled distribution shifts. We further show that the same qualitative behavior persists in modern LLMs and is not mitigated by pure model scaling or scratchpad supervision. Our results highlight a mismatch between training-time skill acquisition and the human-like hierarchical compositions, with implications for reasoning reliability and out-of-distribution robustness.
Figures & tables
| Addition | Multi | Comparison | Sorting | |
| vocab size | 15 | 15 | 18 | 15 |
| max seq len | 21 | 85 | 12 | 40 |
| sample size | 1M | 100K | 45K | 100K |
| Error case 1 | Error case 2 | |
| Swapping | {\color[rgb]{1,0,0}a}_{1}{\color[rgb]{1,0,0}a}_{2}{\color[rgb]{1,0,0}a}_{3}{\color[rgb]{0,0,1}b}_{4} , {\color[rgb]{0,0,1}b}_{1}{\color[rgb]{0,0,1}b}_{2}{\color[rgb]{0,0,1}b}_{3}{\color[rgb]{1,0,0}a}_{4} | {\color[rgb]{1,0,0}a}_{1}{\color[rgb]{1,0,0}a}_{2}{\color[rgb]{0,0,1}b}_{3}{\color[rgb]{0,0,1}b}_{4} , {\color[rgb]{0,0,1}b}_{1}{\color[rgb]{0,0,1}b}_{2}{\color[rgb]{1,0,0}a}_{3}{\color[rgb]{1,0,0}a}_{4} |
| Repeating | {\color[rgb]{1,0,0}a}_{1}{\color[rgb]{1,0,0}a}_{2}{\color[rgb]{1,0,0}a}_{3}{\color[rgb]{0,0,1}b}_{4} , {\color[rgb]{0,0,1}b}_{1}{\color[rgb]{0,0,1}b}_{2}{\color[rgb]{0,0,1}b}_{3}{\color[rgb]{0,0,1}b}_{4} | {\color[rgb]{1,0,0}a}_{1}{\color[rgb]{1,0,0}a}_{2}{\color[rgb]{0,0,1}b}_{3}{\color[rgb]{0,0,1}b}_{4} , {\color[rgb]{0,0,1}b}_{1}{\color[rgb]{0,0,1}b}_{2}{\color[rgb]{0,0,1}b}_{3}{\color[rgb]{0,0,1}b}_{4} |
| Group | Example input | Condition | Mixing Error | Error SD |
| Conflicting | 1000,6 5 8 9 ,6 6 8 2 ,9999 | conflicting in ({\color[rgb]{0,0,1}b}_{2},{\color[rgb]{0,0,1}c}_{2}) and ({\color[rgb]{1,0,0}b}_{4},{\color[rgb]{1,0,0}c}_{4}) | ||
| Agreeing | 1000,6 5 8 2 ,6 6 8 9 ,9999 | agreeing in ({\color[rgb]{0,0,1}b}_{2},{\color[rgb]{0,0,1}c}_{2}) and ({\color[rgb]{0,0,1}b}_{4},{\color[rgb]{0,0,1}c}_{4}) | % |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | 10M (Baseline) | 20M | 100M |
| Layers ( ) | 6 | 12 | 14 |
| Attention Heads ( ) | 6 | 12 | 12 |
| Embedding Dim ( ) | 384 | 384 | 768 |
| Total Parameters | 10M | 20M | 100M |
| Hyperparameter | NanoGPT (Default) | Pythia-1B (Fine-tune) |
| Optimizer | AdamW | AdamW |
| LR Schedule | Cosine | Cosine |
| Learning Rate | ||
| Min Learning Rate | ||
| Weight Decay | 0.1 | 0.1 |
| Betas ( ) | (0.9, 0.98) | (0.9, 0.98) |
| NCID ( ) | Constraint | Generation Logic | Example | Example |
| 0 | No guaranteed match | ; | ||
| 1 | ; others random | |||
| 2 | ; others random | |||
| 3 | ; last digit random | |||
| 4 |
| Length | NCID ( ) | Constraint | Example input |
| 0 | |||
| 1 | & | ||
| 2 | & |
| Asset / official source | License or terms |
| NanoGPT | MIT |
| EleutherAI/pythia-1b | Apache License 2.0 |
| GSM8K | MIT |
| mistralai/Mistral-7B-Instruct-v0.3 | Apache License 2.0 |
| meta-llama/Meta-Llama-3-8B-Instruct | Meta Llama 3 Community License Agreement and Acceptable Use Policy |
| google/gemma-2-9b-it | Gemma Terms of Use , including the Prohibited Use Policy |
| Test set | Construction |
| Hard carry | All operands are three-digit numbers, and the carry at each digit position is maximal. For operands, the carry in each column is required to be . For example, in the two-operand case, has a carry of in each of the ones, tens, and hundreds columns. |
| Close numbers with same prefix | All operands are three-digit numbers sharing the same hundreds and tens digits; only the ones digit varies. For example, . |
| Edge-result sums | The target sum is constrained to lie in , where is a multiple of . For example, , , and . |
| Distribution-shifted test set | 2 operands | 3 operands | 4 operands |
| Hard carry | |||
| Close numbers with same prefix | |||
| Edge-result sums |
| Subskill | Description |
| Subskill 1 | Compare the 1st digit |
| Subskill 2 | Compare the 2nd digit (apply only if Subskill 1 fails to decide). |
| Subskill 3 | Compare the 3rd digit (apply only if Subskills 1-2 fail to decide) |
| Subskill 4 | Compare the 4th digit (apply only if Subskills 1-3 fail to decide) |
| Subskill 5 | If all digits are identical, output ’=’ |
| Set ( ) | Active Position | Constraint | Example | Example | Label |
| 1 | Thousands | ; | |||
| 2 | Hundreds | ; | |||
| 3 | Tens | ; | |||
| 4 | Units | ; |
| Subskill | Description |
| Subskill 1 | Identify and compare the length of each number. |
| Subskill 2 | Compare the 1st digit (apply only if Subskill 1 fails to decide). |
| Subskill 3 | Compare the 2nd digit (apply only if Subskills 1–2 fail to decide). |
| Subskill 4 | Compare the 3rd digit (apply only if Subskills 1–3 fail to decide). |
| Subskill 5 | Compare the 4th digit (apply only if the number is 4-digit and Subskills 1–4 fail to decide). |
| Test file description | Targeted subskill | Criterion |
| Doubly balanced test | Length comparison | All output numbers are of correct lengths |
| NCID = 0, | 1st digit comparison | Output the 1st digits of the 4 input numbers in correct order |
| NCID = 1, | 2nd digit comparison | Output the 2nd digits of the 4 input numbers in correct order |
| NCID = 2, | 3rd digit comparison | Output the 3rd digits of the 4 input numbers in correct order |
| NCID = 3, | 4th digit comparison | Output the 4th digits of the 4 input numbers in correct order |
| NCID ( ) | Constraint | Example input |
| 0 | ||
| 1 | ||
| 2 |
| Test id | Example input | conf. | Label | Mix Err. | SD | ||
| SD1 | 1000,6589,6682,9999 | yes | yes | yes | baseline | 6.63% | 0.46% |
| SD2 | 1000,6589,7682,9999 | no | yes | yes | change | 3.61% | 0.32% |
| SD3 | 1000,6589,6672,9999 | yes | no | yes | change | 0.0% | 0.0% |
| SD4 | 1000,6589,5672,9999 | no | no | yes | both changed | 0.0% | 0.0% |
Numerical Reasoning in Language Models
Numerical reasoning in language models studies how systems represent, calculate with, and explain numbers across mathematical and real-world tasks.
See all 167 papers