Mechanistic interpretability has identified transformer circuits, but lacks a shared vocabulary for describing how their functions compose across tasks and architectures. We introduce Coherentist Probabilistic Compositionalism (CPC), an interpretive framework that grounds transformer computation in coherentist theories of interpretation and describes it through four operator roles. Alignment identifies candidate relations, unification integrates supporting information, suppression reduces incompatible alternatives, and routing carries selected information to the output. Across 15 models from five architecture families, the suppression, unification, and routing weight-space signatures correlate with held-out activation-level role measures above random baselines. Suppression is more stable across tasks than unification. Ablating alignment heads reduces downstream suppressive activity beyond a random-head control in 10 models, but similar effects on no-conflict prompts indicate a general upstream dependency, not contradiction-specific coupling. Explicit contradictions significantly shift a layerwise coherence proxy in 14 models; after removing shared residual covariance, the gap has the predicted direction in every model. Base and instruction-tuned variants preserve induction-head score structure (r≥0.98) without a consistent shift of operator signatures towards later layers. These results support CPC as a shared vocabulary for comparing transformer mechanisms while showing that their depth and geometric expression remain architecture-specific.
Figures & tables
Figure 1: The CPC vocabulary on the give-frame running example. Partial cues enter, alignment proposes candidate role links, unification integrates the compatible assignments, suppression prunes the incompatible recipient, and routing conveys the consistent interpretation to the readout, with coherence increasing across depth. The legend names a characteristic mechanism for each operator, and the same operators also cover factual recall, contradiction handling, and instruction tuning. Individual mechanisms may instantiate one or more operator roles.
Figure 2: A coherence state and its groundings. Fragments m1 - m4 capture partial hypotheses about spans of the input, grey lines mark the positions Um they cover, and dashed teal groundings map fragment positions into the relational graph G .
Figure 3: Coherence as a representational configuration. Circles are grounded fragments, fill level shows support, teal links mark mutual support, and red dashed links mark unresolved incompatibility, not disagreement. Refinement increases coverage and compatibility and decreases conflict, raising C(A) (Section 2 ; 7 ; 64 ). All fragments can be retained provided their relations are admissible, and support stays graded.
Figure 4: The CPC operators on the transformer architecture, with tokens on the left and depth running left to right. Query-key structure aligns positions, OV and feedforward writes unify content, subtractive writes suppress incompatible content, and a mover head routes the result to the readout.
Operator
Mechanistic signature
Representative circuits
Alignment
QK compatibility structure
Previous-token, duplicate-token heads
Unification
Positive OV or feedforward writes
Induction-head OV; key-value memories
Suppression
Negative OV modes; subtractive writes
S-inhibition, copy-suppression heads
Routing
Content-preserving OV structure; positional transport (activations)
Name-mover heads
Table 1: CPC operators, their candidate mechanistic signatures, and representative circuits associated with each role; sources are cited in the text.
Model
Sil.
Best k
pshuf
W → Act r
Magn. r
Bl.
Abl.
GPT-2 Small
0.42
3
0.15
0.54
0.22
3/3
–
GPT-2 Medium
0.46
3
<0.01
0.54
0.11
3/3
✓
GPT-2 Large
0.42
3
<0.01
0.58
0.15
3/3
✓
Pythia-410M
0.36
5
0.82
0.30
0.07
3/3
✓
Pythia-1.4B
0.42
3
<0.01
0.34
0.05
3/3
–
Pythia-2.8B
0.36
6
<0.01
0.33
0.10
3/3
–
Table 2: Operator-signature structure and functional validation (means over 5 seeds), computed on the four independent signature coordinates (alignment, suppression, unification, routing-copy). Sil.: silhouette at k=4 ; Best k : silhouette-optimal cluster count; pshuf : maximum column-shuffle null p -value across seeds; W → Act r : held-out Spearman correlation between weight scores and activation-level role measures, against the weight-magnitude baseline (Magn.); Bl.: baselines beaten (random-projection, shuffled-relation, weight-magnitude); Abl.: positive rank correlation with held-out single-head ablation effects. The pre-specified P1 verdict is support with 3/3 baselines and a positive ablation correlation (three models), partial otherwise (twelve).
Figure 5: Causal ablation of operator-ranked heads on held-out IOI (mean over 5 seeds). Filled dots: change in the correct-token logit when zero-ablating the top five heads per operator ranking (routing ranked by the embedding-space copy score, Equation 6 ); open diamonds: layer-matched random-head control. Dots left of zero indicate removed support for the correct answer.
Model
Δ contra
Δ rand.
Δ no-conf.
P2
GPT-2 Small
-1.39 ∗
-0.43
-1.74
part.
GPT-2 Medium
+0.02
-0.01
-0.02
agai.
GPT-2 Large
-0.01 ∗
-0.00
-0.02
part.
Pythia-410M
-0.21 ∗
-0.01
-0.33
part.
Pythia-1.4B
-0.10 ∗
-0.05
-0.14
part.
Pythia-2.8B
-0.34 ∗
+0.01
-0.44
part.
Table 3: Suppression-alignment coupling (mean over 5 seeds): change in summed suppressive contribution when ablating alignment heads on contradiction prompts ( ∗ : exceeds the layer-matched random-head control, Holm-corrected p<0.05 ), the random-head control, and the same intervention on matched no-conflict prompts. P2: pre-specified verdict ( supp. / part. / agai. ).
Figure 6: Layerwise coherence gap (consistent minus contradictory) for all 15 models. Lines give the mean over 5 seeds, bands the range across seeds, and colour the sign of the mean gap (navy positive, orange negative). GPT-2 and Pythia show the predicted positive gap, Gemma, Qwen, and three of the four LLaMA variants show negative gaps, and LLaMA-1B-Instruct follows the positive families.
Pair
Late s+r
Early a
Ratio
Ind. r
Pres.
Verdict
Gemma-2B
0.006
0.017
0.33
0.982
yes
partial
LLaMA-1B
0.060
0.051
1.18
0.992
yes
support
LLaMA-3B
0.055
0.040
1.36
0.991
yes
support
Qwen-1.5B
0.002
0.003
0.61
0.998
yes
partial
Table 4: Base vs. instruction-tuned reweighting (mean over 5 seeds): late-half suppression-and-routing signature shift, early-half alignment shift, their ratio, the induction-head correlation between variants, and the outcome of the one-sided preservation test (threshold r=0.9 ). Verdicts follow the pre-specified P4 rule.