Linear Probes to Bayesian Manifolds: Geometry of Arithmetic in Large Language Models
Organizations: Carnegie Mellon University · Indian Institute of Technology Jodhpur · Indian Institute of Technology Roorkee
Abstract
Linear probes are a standard method to study geometric structures, such as helices and circles, from the activations of large language models (LLMs), and such structure is often taken as evidence that the model computes with it. Probe accuracy alone cannot establish this, because it conflates four separate claims. These are (1) that the concept is decodable from the activations, (2) that they form a specific shape, (3) that the shape belongs to the probed concept rather than to a related concept the model also represents, and (4) that the output of the model depends on the shape. We call these claims decodability, shape, ownership, and causal use, and we test them separately with a four-stage Bayesian pipeline. Stage 2 compares candidate shapes and selects the best one only when its evidence passes a predefined threshold. Stage 3 removes a set of related concepts, specified in advance, and tests whether the shape is owned or inherited. Stage 4 intervenes on the shape to test whether the output of the model depends on it. We apply the pipeline to two-digit addition and multiplication in GPT-J 6B, Llama 3.1 8B, and Pythia 6.9B. Nearly every concept is decodable, but the decoded directions overlap substantially across concepts. Every strongly supported comparison selects a ribbon, which is a helix whose radius changes with the value, or a torus. Of 1,330 cells with an ownership verdict, 19% of shapes are owned and 81% are inherited. Owned and inherited shapes move the output about equally often. Intervening on the shape retains approximately 61% of the effect of intervening on the full probe while using at most five dimensions, against a median of nine for the probe. Our results indicate that a probe verdict alone overstates what a model computes with, and we propose that probe results be reported as verdicts on all four claims rather than as a single accuracy.
Figures & tables
| Stage 1 | Stage 2, shape | Stage 3, ownership | Stage 4, ablation | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Task | Superposition | Ribbon | Torus | Other | Owned | Inherited | Retention | ||
| GPT-J | addition | 0.808 | 161 | 42 | 0 | 24 | 147 | 0.70 | ||
| GPT-J | multiplication | 0.897 | 238 | 43 | 7 | 50 | 199 | 0.68 | ||
| Llama | addition | 0.782 | 173 | 37 | 1 | 42 | 151 | 0.32 | ||
| Llama | multiplication | 0.907 | 256 | 19 | 5 | 46 | 226 | 0.55 | ||
| Pythia | addition | 0.815 | 141 | 61 | 3 | 42 | 149 | 0.43 | ||
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning | Example |
|---|---|---|
| , | The two operands, integers from 0 to 99 | 23, 17 |
| , | The sum and the product, which are the answers in addition and in multiplication | product 391 |
| , | Units and tens digits of , equal to and . The same subscripts apply to , and the tens digit of an answer is | 3, 2 |
| Indicator, equal to 1 when the condition holds and 0 otherwise | ||
| The real numbers | ||
| A model, one of GPT-J 6B, Llama 3.1 8B, and Pythia 6.9B |
| Symbol | Meaning | Example |
|---|---|---|
| Mean activation of the problems of a cell | ||
| , , , ⊤ | The identity matrix, the vectors of zeros and of ones, and the transpose | |
| Gaussian distribution with mean and covariance . Its density at is | ||
| , | Activation and concept value of problem | |
| Rank of the probe subspace, the number of singular vectors kept in Stage 1 | ||
| Orthonormal basis of the probe subspace found in Stage 1 |
| Symbol | Meaning | Example |
|---|---|---|
| Number of values of a concept that remain in a cell | 10 | |
| Weighted and centred centroids, one row per value | ||
| , | Squared singular value of rank of , and the percentile of the same quantity under permutation | |
| , | Ratio of the residual dimension to , and trace of the residual covariance over the residual dimension, in the Marčenko–Pastur check | |
| , | Parts of the activations outside , and the largest eigenvalue of their covariance | |
| Marčenko–Pastur edge, the largest eigenvalue that noise alone produces outside |
| Abbreviation | Full term |
|---|---|
| BPE | Byte pair encoding, the tokenisation scheme of the three models |
| CPU | Central processing unit |
| FDR | False discovery rate |
| LDA | Linear discriminant analysis |
| LLM | Large language model |
| OAS | Oracle approximating shrinkage, a covariance estimator |
| Term | Meaning |
|---|---|
| Cell | One combination of model, task, layer, concept, and preprocessing setting |
| Candidate cell | A cell of one setting before any exclusion rule applies |
| Cell with a shape fit | A cell in which Step 2c computes the evidence of every candidate shape |
| Core concepts | The six concepts per task that are also reported separately |
| Extended concepts | All other listed concepts |
| Joint concept | A concept formed from two or three single concepts |
| Model | Addition | Multiplication |
|---|---|---|
| GPT-J 6B | 8,415 (84.15%) | 2,751 (91.00%) |
| Llama 3.1 8B | 9,963 (99.63%) | 2,927 (96.82%) |
| Pythia 6.9B | 7,718 (77.18%) | 2,757 (91.20%) |
| Task | Concept | Related concepts |
|---|---|---|
| Addition | sum | , |
| Addition | units digit of the answer | , , , , units column sum, units carry |
| Addition | tens digit of the answer | , , , , tens column sum, units carry, tens carry |
| Multiplication | units carry | units column sum, units partial product |
| Multiplication | units digit of the answer | , , , , units partial product, units column sum, units carry |
| Concept | Related concepts |
|---|---|
| a | a_units , a_tens |
| b | b_units , b_tens |
| a_units , a_tens | a |
| b_units , b_tens | b |
| answer | a , b |
| ans_units | a , b , a_units , b_units , column_sum_units , carry_units |
| Concept | Related concepts |
|---|---|
| a | a_units , a_tens |
| b | b_units , b_tens |
| a_units , a_tens | a |
| b_units , b_tens | b |
| answer | a , b |
| ans_units | a , b , a_units , b_units , partial_product_units , column_sum_units , carry_units |
| Group | Concepts |
|---|---|
| Core (6) | a , b , answer , ans_units , ans_tens , carry_units |
| Extended single (35) | a_eq_b , a_is_zero , a_magnitude_tier , a_num_digits , a_parity , a_tens , a_units , ans_ends_in_zero , ans_hundreds , ans_is_zero , ans_magnitude_tier , ans_num_digits , ans_parity , b_is_zero , b_magnitude_tier , b_num_digits , b_parity , b_tens , b_units , both_one , both_zero , carry_tens , column_sum_tens , column_sum_units , either_one , either_zero , larger_operand , max_operand , min_operand , operand_abs_diff , operand_diff , parity_match , parity_xor , running_sum_tens , running_sum_units |
| Joint (10) | a_tens__b_tens , a_tens__b_units , a_units__b_tens , a_units__b_units , carry_units__ans_units , carry_units__column_sum_units , a_tens__b_tens__ans_tens , a_tens__b_tens__carry_units , a_units__b_units__ans_tens , a_units__b_units__ans_units |
| Group | Concepts |
|---|---|
| Core (6) | a_units , a_tens , b_units , partial_product_units , carry_units , ans_units |
| Extended single (41) | a , a_eq_b , a_is_zero , a_magnitude_tier , a_num_digits , a_parity , ans_ends_in_zero , ans_hundreds , ans_is_zero , ans_magnitude_tier , ans_num_digits , ans_parity , ans_tens , answer , b , b_is_zero , b_magnitude_tier , b_num_digits , b_parity , b_tens , both_one , both_zero , carry_tens , column_sum_hundreds , column_sum_tens , column_sum_units , either_one , either_zero , larger_operand , max_operand , min_operand , operand_abs_diff , operand_diff , parity_match , parity_xor , partial_product_a_tens_b_tens , partial_product_a_tens_b_units , partial_product_a_units_b_tens , running_sum_hundreds , running_sum_tens , running_sum_units |
| Joint (10) | a_tens__b_tens , a_tens__b_units , a_units__b_tens , a_units__b_units , carry_units__ans_units , carry_units__column_sum_units , a_tens__b_tens__ans_tens , a_tens__b_tens__carry_units , a_units__b_units__ans_tens , a_units__b_units__partial_product_units |
| Population | Setting | Condition | Cells |
|---|---|---|---|
| All cells | all three | every model, task, layer, concept, and setting | 4,860 |
| Candidate cells | one | before any exclusion rule | 1,620 |
| Removed by the floor | off | fewer than two values with 30 problems | 155 |
| Removed by the permutation test | off | no direction kept, | 20 |
| Decodable cells | off | 1,445 | |
| Cells with a shape fit | pooled | evidence of every candidate shape computed | 1,457 |
| Shape | Features | Description |
|---|---|---|
| generic baseline | smooth trend with no periodic feature | |
| line | values ordered along one direction | |
| circle | values repeat with period | |
| open helix | circle that advances along a line | |
| torus | product of two circles | |
| concentric | circle with a fixed radial drift |
| Quantity | Value |
|---|---|
| Stage 1 | |
| Minimum number of problems per value | 30 |
| Flag for imbalance, most against least frequent value | more than 3 times |
| Permutations for the rank of the probe subspace, and level | 1,000, 99th percentile |
| Permutations and folds for the LDA directions | 1,000, 5 folds |
| Ledoit–Wolf shrinkage of the total scatter | when |
| Superposition | Other shapes | Patching | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Task | SD | Mean | Maximum | Baseline | Circle | Open helix | ||
| GPT-J | addition | 0.046 | 6.0 | 11.9 | 0 | 0 | 0 | ||
| GPT-J | multiplication | 0.014 | 10.0 | 17.3 | 4 | 2 | 1 | ||
| Llama | addition | 0.027 | 7.8 | 27.4 | 1 | 0 | 0 | ||
| Llama | multiplication | 0.020 | 8.9 | 13.0 | 5 | 0 | 0 | ||
| Pythia | addition | 0.046 | 6.9 | 14.1 | 3 | 0 | 0 | ||
| off | answer | norm | all | |
| Tier A, exact shape | 100 | 118 | 120 | 338 |
| Tier A, family | 31 | 14 | 36 | 81 |
| Tier B | 348 | 245 | 318 | 911 |
| Cells with an ownership verdict | 479 | 377 | 474 | 1,330 |
| Ribbon has the highest evidence | 399 | 314 | 394 | 1,107 |
| Torus has the highest evidence | 78 | 61 | 78 | 217 |
| Kind of concept | Verdicts | Owned | Pass all four |
|---|---|---|---|
| Functions of the operands | 252 | 108 | 10 |
| Running sums | 180 | 49 | 7 |
| Operand digits | 298 | 33 | 2 |
| Column sums | 151 | 30 | 7 |
| Operands | 124 | 25 | 17 |
| Partial products | 98 | 5 | 3 |
| Addition | ||||||
|---|---|---|---|---|---|---|
| Model | Answer units | Answer tens | Sum | |||
| GPT-J | ribbon | ribbon | torus | torus | ribbon | |
| Llama | ribbon | ribbon | ribbon | ribbon | ribbon | |
| Pythia | ribbon | baseline | torus | torus | ribbon | |
| Multiplication | ||||||
| Model | Units of | Tens of | Units of | Partial product | Carry | Answer units |
| Layer | ||||||
|---|---|---|---|---|---|---|
| Model | Task | 4 | 8 | 14 or 16 | 20 or 24 | 24 or 28 |
| GPT-J | addition | / | / | / | / | / |
| GPT-J | multiplication | / | / | / | / | / |
| Llama | addition | / | / | / | / | / |
| Llama | multiplication | / | / | / | / | / |
| Pythia | addition | / | / | / | / | / |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 837 papers