Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also raises a robustness concern: models may reuse irrelevant tokens from earlier context when answering a later query. Prior work identifies this behavior as contextual entrainment and finds attention heads whose ablation reduces it, arguing that these heads are largely task- or relation-specific. But are the mechanisms behind contextual entrainment truly isolated by relation, or do relation-conditioned searches recover reusable components shared across relations? Using factual-distractor relation-completion prompts across 19 lexical-semantic subtypes, we identify attention heads whose ablation reduces entrainment and test how the resulting head sets transfer recovery ability across exact relations, semantic families, and mixed relation distributions. Same-family transfer exceeds different-family transfer, family residuals weaken once globally shared heads are removed, and subtype-specific searches repeatedly rediscover many global heads. Clean-completion and context-use probes further suggest that this shared intervention can reduce harmful copying while preserving ordinary relation completion and simple in-context abilities. Our findings provide a more precise perspective on contextual entrainment and its mitigation: harmful context following is not best addressed by searching for isolated relation-specific circuits, but by using heterogeneous attack prompts to reveal reusable intervention targets whose effects are shaped by semantic relation structure.
Figures & tables
Figure 1: Factual-distractor examples across four lexical-semantic families.
Family
Sub.
Pairs
Valid test
Rate
Dˉ
Association
4
163
441/480
91.9
3.92
Attribute
5
244
293/600
48.8
2.69
Part–Whole
5
224
492/600
82.0
4.77
Taxonomy
5
227
496/600
82.7
3.60
Total
19
858
1722/2280
75.5
3.86
Table 1: Benchmark statistics by semantic family after tokenizer and valid-attack filtering. Valid test uses the main filter D≥0.5 ; Dˉ is the mean attack drop.
Figure 2: Transfer is graded by semantic relatedness. The top panel illustrates the transfer conditions from a source circuit to target relations. The bottom panel reports target-centered recovery, which controls for target-specific recoverability.
Circuit
Assoc.
Attr.
P–W
Tax.
Avg.
Global shared
1.502
1.801
1.028
1.531
1.464
Family shared
1.486
1.904
0.925
0.943
1.306
Family residual
0.881
1.243
0.699
0.997
0.959
Subtype-specific
1.606
1.619
1.069
1.361
1.404
Table 2: Shared-circuit decomposition. Values are normalized recovery. Family residuals exclude global-shared top- K heads from the family ranking.
Head set
K=20
K=40
K=80
Global search
0.207
0.724
1.464
Frequency-defined global
0.364
0.639
1.350
Frequency ∩ global
0.361
0.788
1.431
Subtype-specific
0.301
0.654
1.404
Family-shared
0.204
0.551
1.306
Rare heads
0.004
0.014
0.082
Table 3: Frequency-defined global heads recover most of the global circuit’s effect, while rarely rediscovered heads do not. Values are normalized recovery.
Figure 3: External LRE clean relation-completion capability after ablation.
Figure 4: Comparison among single-relation, general-relation and mixed-relation search.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Subtype
Example pair
Representative template
Association
country–capital
France → Paris
The capital of France is Paris.
Association
profession–workplace
doctor → hospital
The workplace of a doctor is a hospital.
Association
sport–equipment
tennis → racket
The equipment used in tennis is a racket.
Association
task–tool
writing → pen
The tool used for writing is a pen.
Attribute
color
banana → yellow
The typical color of a banana is yellow.
Attribute
shape
coin → round
The typical shape of a coin is round.
Appendix
Table 4: Relation subtypes and representative templates in the lexical-semantic benchmark.
Subtype
Template 1
Template 2
country–capital
The capital of {x} is {y}.
The capital city of {x} is {y}.
profession–workplace
The workplace of a {x} is a {y}.
A {x} typically works in a {y}.
sport–equipment
The equipment used in {x} is a {y}.
People use a {y} in {x}.
task–tool
The tool used for {x} is a {y}.
People use a {y} for {x}.
color
The typical color of a {x} is {y}.
The color commonly associated with a {x} is {y}.
shape
The typical shape of a {x} is {y}.
The shape commonly associated with a {x} is {y}.
Appendix
Table 5: Prompt templates used for each relation subtype.
Threshold
Valid tests
Same-family
Diff.-family
Raw adv.
Centered adv.
p
D≥0.25
1776
0.997
0.871
0.125
0.128
0.0022
D≥0.50
1722
0.956
0.838
0.118
0.123
0.0021
D≥0.75
1660
0.926
0.812
0.113
0.118
0.0021
D≥1.00
1590
0.885
0.779
0.106
0.112
0.0024
Appendix
Table 6: Valid-attack threshold sensitivity. Same-family transfer remains stronger than different-family transfer across all tested attack-drop thresholds. Same-family and different-family columns report non-target-centered normalized recovery. The centered advantage is computed after target-centering; p is the cell-level Welch test for the centered same-family versus different-family comparison.
Statistic
Observed
Median
95% CI
p
Same-family advantage
0.123
0.115
[0.003, 0.236]
0.024
Appendix
Table 7: Two-way clustered bootstrap over source and target subtypes. The statistic is the target-centered same-family minus different-family advantage.
Method
Diff.-family
Same-family
Same-subtype
Gumbel
0.693 (2.711)
1.081 (3.737)
1.407 (5.418)
Random
-0.167 (-0.695)
-0.089 (-0.395)
-0.123 (-0.458)
Same-layer random
-0.202 (-0.659)
-0.117 (-0.349)
-0.213 (-0.818)
Appendix
Table 8: Random and same-layer random controls.
Transfer condition
Target-centered recovery
Raw Δ
Different-family
-0.054
2.80
Same-family
+0.069
3.16
Same-subtype
+0.512
4.72
Appendix
Table 9: Target-centered transfer by semantic relatedness. Target-centering removes target-specific recoverability before comparing source circuits.
Metric
Mistral-7B
Qwen2.5-7B
Global heads rediscovered
79/80
79/80
Uniform expected overlap
6.25
8.16
Global–family mean overlap
40.75
46.25
Additive R2
0.523
0.553
Full-interaction R2
0.528
0.560
Prediction correlation
0.995
0.994
Appendix
Table 10: Compact summary of the main cross-model replication results under the same protocol. Overlap rows compare observed head-set overlap against the within-model uniform-head baseline; the final three rows summarize the additive decomposition analysis.
Model
Diff.-family
Same-family
Same-subtype
LLaMA-3.1-8B
1.000
1.141
1.675
Mistral-7B
1.000
1.239
1.823
Qwen2.5-7B
1.000
1.229
1.614
Appendix
Table 11: Cross-model transfer summary. Values are non-target-centered normalized recovery under valid-attack filtering. Row-wise normalization by the different-family value is used to reduce apparent differences induced by model-specific scale.
Circuit
Top-3
Top-10
Base
82.90
92.86
Global shared
84.07
92.86
Family-shared avg.
82.17
92.89
Subtype-specific avg.
78.50
91.72
Subtype-specific: country–capital
72.25
90.87
Appendix
Table 12: External LRE clean capability. Global-shared ablation preserves clean relation completion, while subtype-specific circuits cause larger collateral degradation on average.
Relation
Top-3
Top-10
Base
Global
Subtype
Base
Global
Subtype
fruit outside color
40.0
56.7
13.3
86.7
83.3
86.7
landmark in country
90.0
91.5
88.0
95.5
96.0
95.5
landmark on continent
84.5
84.0
76.5
97.0
97.5
97.0
product by company
85.5
86.0
84.0
93.5
93.0
91.0
star constellation name
85.6
87.8
41.7
89.2
89.9
80.6
Appendix
Table 13: External LRE clean capability test (full per-relation breakdown; corresponds to Figure 3 in the main text). Base denotes no ablation; Global denotes the mixed-relation global circuit; Subtype denotes the country–capital subtype-specific circuit.
Circuit
n
K
Top-1
Top-3
Top-10
Δ Top-3
Δ Top-10
Base
272
0
31.62
39.71
66.91
0.00
0.00
Global shared
272
80
31.62
38.60
61.03
1.10
5.88
Appendix
Table 14: Expanded context-use preservation probe. Accuracies and drops are in percentage points; positive drops mean worse than the base model.
Figure 5: Activation replacement robustness. Clean-to-attacked activation replacement recovers substantial attacked margin for global and frequency-defined global heads, while rare heads remain weak. Zero ablation is stronger, but the replacement result shows that the global component is not merely a zero-ablation artifact.
Term
Additive
Full interaction
Intercept
−0.254
−0.001
G
2.936
2.492
F
1.009
0.735
S
1.159
0.886
G:F
–
0.425
G:S
–
0.422
Appendix
Table 15: Factorial regression on raw margin recovery; interaction terms add little beyond the additive model.
Condition
Recovery
Raw Δ
Clean harm
G40+F20+S20
1.550
5.003
−1.012
G80 global only
1.464
4.900
−0.990
S80 subtype only
1.404
4.715
−1.073
F80 family only
1.306
4.254
−1.111
Random 80
−0.091
−0.308
0.523
Same-layer random 80
−0.321
−1.000
0.664
Appendix
Table 16: Fixed-budget diagnostic (80 heads): the structured mix slightly outperforms single-component circuits; random controls fail.
Figure 6: An illustrative factual-distractor example under the ranking-based intervention used in our Gumbel-Sigmoid head search. The clean prompt favors crew ; the distractor context shifts probability toward council ; and zero-ablating the top-80 heads exported from the learned mask scores reverses this shift.
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.
Larger language models become simultaneously better and worse at handling contextual information -- better at ignoring false claims, worse at ignoring irrelevant tokens. We formalize this apparent paradox through the first scaling laws for contextual entrainment, the tendency of models to favor tokens that appeared in context regardless of relevance. Analyzing the Cerebras-GPT (111M-13B) and Pythia (410M-12B) model families, we find entrainment follows predictable power-law scaling, but with opposite trends depending on context type: semantic contexts show decreasing entrainment with scale, while non-semantic contexts show increasing entrainment. Concretely, the largest models are four times more resistant to counterfactual misinformation than the smallest, yet simultaneously twice as prone to copying arbitrary tokens. These diverging trends, which replicate across model families, suggest that semantic filtering and mechanical copying are functionally distinct behaviors that scale in opposition -- scaling alone does not resolve context sensitivity, it reshapes it.
Dikshant Kukreja, Kshitij Sah, Gautam Gupta +5
IIIT Delhi, India · Singapore Institute of Technology · NVIDIA +1
Large language models reuse the same attention heads across tasks, but identifying these shared heads does not explain what they do differently in each task. A natural source of inspiration is task-dependent interactions among shared brain regions, which motivate a network view of language-model computation. Here, we compare fixed model components across tasks using identical graph facts, statements specifying which directed connections are present. We measure how interventions affect attention heads and the activation state in later layers (influence), as well as the answer (importance). Task changes redistribute the influence in five open-weight models spanning four families and 1.5-14 billion parameters. In Gemma-2-9B and Qwen3-14B, an activation edit changes an early internal activation vector and switches which fact the model reads while the written instruction stays fixed. The same heads then respond to different graph facts and influence later components differently. For each prompt, we construct two linear response maps, one before the edit and one after it. Each map predicts a later head's response from a measured change in an earlier head. For a separate factual intervention, the map for the corresponding condition reduces mean squared prediction error by 53.6% in Gemma and 5.0% in Qwen compared with the average of the two maps. These findings show that task context changes both which graph facts the shared components respond to and how their outputs affect later computation. A network account may help explain how models use shared parameters to perform different computations. Measurements of task-dependent influence could also guide interventions intended to alter one behavior while preserving others.