From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

Jinghan Zheng; Jiajun Shi

cs.CLOct 13, 2026

From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

Authors: Jinghan Zheng, Jiajun Shi

Organizations: Dalian University of Technology

Abstract

Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also raises a robustness concern: models may reuse irrelevant tokens from earlier context when answering a later query. Prior work identifies this behavior as contextual entrainment and finds attention heads whose ablation reduces it, arguing that these heads are largely task- or relation-specific. But are the mechanisms behind contextual entrainment truly isolated by relation, or do relation-conditioned searches recover reusable components shared across relations? Using factual-distractor relation-completion prompts across 19 lexical-semantic subtypes, we identify attention heads whose ablation reduces entrainment and test how the resulting head sets transfer recovery ability across exact relations, semantic families, and mixed relation distributions. Same-family transfer exceeds different-family transfer, family residuals weaken once globally shared heads are removed, and subtype-specific searches repeatedly rediscover many global heads. Clean-completion and context-use probes further suggest that this shared intervention can reduce harmful copying while preserving ordinary relation completion and simple in-context abilities. Our findings provide a more precise perspective on contextual entrainment and its mitigation: harmful context following is not best addressed by searching for isolated relation-specific circuits, but by using heterogeneous attack prompts to reveal reusable intervention targets whose effects are shaped by semantic relation structure.

Figures & tables

Figure 1: Factual-distractor examples across four lexical-semantic families.

Family	Sub.	Pairs	Valid test	Rate	$\mathbf{\bar{D}}$
Association	4	163	441/480	91.9	3.92
Attribute	5	244	293/600	48.8	2.69
Part–Whole	5	224	492/600	82.0	4.77
Taxonomy	5	227	496/600	82.7	3.60
Total	19	858	1722/2280	75.5	3.86

Table 1: Benchmark statistics by semantic family after tokenizer and valid-attack filtering. Valid test uses the main filter

D\geq 0.5

;

\bar{D}

is the mean attack drop.

Figure 2: Transfer is graded by semantic relatedness. The top panel illustrates the transfer conditions from a source circuit to target relations. The bottom panel reports target-centered recovery, which controls for target-specific recoverability.

Circuit	Assoc.	Attr.	P–W	Tax.	Avg.
Global shared	$1.502$	$1.801$	$1.028$	$1.531$	$1.464$
Family shared	$1.486$	$1.904$	$0.925$	$0.943$	$1.306$
Family residual	$0.881$	$1.243$	$0.699$	$0.997$	$0.959$
Subtype-specific	$1.606$	$1.619$	$1.069$	$1.361$	$1.404$

Table 2: Shared-circuit decomposition. Values are normalized recovery. Family residuals exclude global-shared top-

K

heads from the family ranking.

Head set	$K=20$	$K=40$	$K=80$
Global search	$0.207$	$0.724$	$1.464$
Frequency-defined global	$0.364$	$0.639$	$1.350$
Frequency $\cap$ global	$0.361$	$0.788$	$1.431$
Subtype-specific	$0.301$	$0.654$	$1.404$
Family-shared	$0.204$	$0.551$	$1.306$
Rare heads	$0.004$	$0.014$	$0.082$

Table 3: Frequency-defined global heads recover most of the global circuit’s effect, while rarely rediscovered heads do not. Values are normalized recovery.

Figure 3: External LRE clean relation-completion capability after ablation.

Figure 4: Comparison among single-relation, general-relation and mixed-relation search.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Subtype	Example pair	Representative template
Association	country–capital	France $\rightarrow$ Paris	The capital of France is Paris.
Association	profession–workplace	doctor $\rightarrow$ hospital	The workplace of a doctor is a hospital.
Association	sport–equipment	tennis $\rightarrow$ racket	The equipment used in tennis is a racket.
Association	task–tool	writing $\rightarrow$ pen	The tool used for writing is a pen.
Attribute	color	banana $\rightarrow$ yellow	The typical color of a banana is yellow.
Attribute	shape	coin $\rightarrow$ round	The typical shape of a coin is round.

Appendix

Table 4: Relation subtypes and representative templates in the lexical-semantic benchmark.

Subtype	Template 1	Template 2
country–capital	The capital of {x} is {y}.	The capital city of {x} is {y}.
profession–workplace	The workplace of a {x} is a {y}.	A {x} typically works in a {y}.
sport–equipment	The equipment used in {x} is a {y}.	People use a {y} in {x}.
task–tool	The tool used for {x} is a {y}.	People use a {y} for {x}.
color	The typical color of a {x} is {y}.	The color commonly associated with a {x} is {y}.
shape	The typical shape of a {x} is {y}.	The shape commonly associated with a {x} is {y}.

Appendix

Table 5: Prompt templates used for each relation subtype.

Threshold	Valid tests	Same-family	Diff.-family	Raw adv.	Centered adv.	$p$
$D\geq 0.25$	1776	0.997	0.871	0.125	0.128	0.0022
$D\geq 0.50$	1722	0.956	0.838	0.118	0.123	0.0021
$D\geq 0.75$	1660	0.926	0.812	0.113	0.118	0.0021
$D\geq 1.00$	1590	0.885	0.779	0.106	0.112	0.0024

Appendix

Table 6: Valid-attack threshold sensitivity. Same-family transfer remains stronger than different-family transfer across all tested attack-drop thresholds. Same-family and different-family columns report non-target-centered normalized recovery. The centered advantage is computed after target-centering;

p

is the cell-level Welch test for the centered same-family versus different-family comparison.

Statistic	Observed	Median	95% CI	$p$
Same-family advantage	0.123	0.115	[0.003, 0.236]	0.024

Appendix

Table 7: Two-way clustered bootstrap over source and target subtypes. The statistic is the target-centered same-family minus different-family advantage.

Method	Diff.-family	Same-family	Same-subtype
Gumbel	0.693 (2.711)	1.081 (3.737)	1.407 (5.418)
Random	-0.167 (-0.695)	-0.089 (-0.395)	-0.123 (-0.458)
Same-layer random	-0.202 (-0.659)	-0.117 (-0.349)	-0.213 (-0.818)

Appendix

Table 8: Random and same-layer random controls.

Transfer condition	Target-centered recovery	Raw $\Delta$
Different-family	-0.054	2.80
Same-family	+0.069	3.16
Same-subtype	+0.512	4.72

Appendix

Table 9: Target-centered transfer by semantic relatedness. Target-centering removes target-specific recoverability before comparing source circuits.

Metric	Mistral-7B	Qwen2.5-7B
Global heads rediscovered	79/80	79/80
Uniform expected overlap	6.25	8.16
Global–family mean overlap	40.75	46.25
Additive $R^{2}$	0.523	0.553
Full-interaction $R^{2}$	0.528	0.560
Prediction correlation	0.995	0.994

Appendix

Table 10: Compact summary of the main cross-model replication results under the same protocol. Overlap rows compare observed head-set overlap against the within-model uniform-head baseline; the final three rows summarize the additive decomposition analysis.

Model	Diff.-family	Same-family	Same-subtype
LLaMA-3.1-8B	1.000	1.141	1.675
Mistral-7B	1.000	1.239	1.823
Qwen2.5-7B	1.000	1.229	1.614

Appendix

Table 11: Cross-model transfer summary. Values are non-target-centered normalized recovery under valid-attack filtering. Row-wise normalization by the different-family value is used to reduce apparent differences induced by model-specific scale.

Circuit	Top-3	Top-10
Base	82.90	92.86
Global shared	84.07	92.86
Family-shared avg.	82.17	92.89
Subtype-specific avg.	78.50	91.72
Subtype-specific: country–capital	72.25	90.87

Appendix

Table 12: External LRE clean capability. Global-shared ablation preserves clean relation completion, while subtype-specific circuits cause larger collateral degradation on average.

Relation	Top-3			Top-10
	Base	Global	Subtype	Base	Global	Subtype
fruit outside color	40.0	56.7	13.3	86.7	83.3	86.7
landmark in country	90.0	91.5	88.0	95.5	96.0	95.5
landmark on continent	84.5	84.0	76.5	97.0	97.5	97.0
product by company	85.5	86.0	84.0	93.5	93.0	91.0
star constellation name	85.6	87.8	41.7	89.2	89.9	80.6

Appendix

Table 13: External LRE clean capability test (full per-relation breakdown; corresponds to Figure 3 in the main text). Base denotes no ablation; Global denotes the mixed-relation global circuit; Subtype denotes the country–capital subtype-specific circuit.

Circuit	$n$	$K$	Top-1	Top-3	Top-10	$\Delta$ Top-3	$\Delta$ Top-10
Base	272	0	31.62	39.71	66.91	0.00	0.00
Global shared	272	80	31.62	38.60	61.03	1.10	5.88

Appendix

Table 14: Expanded context-use preservation probe. Accuracies and drops are in percentage points; positive drops mean worse than the base model.

Figure 5: Activation replacement robustness. Clean-to-attacked activation replacement recovers substantial attacked margin for global and frequency-defined global heads, while rare heads remain weak. Zero ablation is stronger, but the replacement result shows that the global component is not merely a zero-ablation artifact.

Term	Additive	Full interaction
Intercept	$-0.254$	$-0.001$
$G$	$2.936$	$2.492$
$F$	$1.009$	$0.735$
$S$	$1.159$	$0.886$
$G{:}F$	–	$0.425$
$G{:}S$	–	$0.422$

Appendix

Table 15: Factorial regression on raw margin recovery; interaction terms add little beyond the additive model.

Condition	Recovery	Raw $\Delta$	Clean harm
$G_{40}+F_{20}+S_{20}$	$1.550$	$5.003$	$-1.012$
$G_{80}$ global only	$1.464$	$4.900$	$-0.990$
$S_{80}$ subtype only	$1.404$	$4.715$	$-1.073$
$F_{80}$ family only	$1.306$	$4.254$	$-1.111$
Random 80	$-0.091$	$-0.308$	$0.523$
Same-layer random 80	$-0.321$	$-1.000$	$0.664$

Appendix

Table 16: Fixed-budget diagnostic (80 heads): the structured mix slightly outperforms single-component circuits; random controls fail.

Figure 6: An illustrative factual-distractor example under the ranking-based intervention used in our Gumbel-Sigmoid head search. The clean prompt favors crew ; the distractor context shifts probability toward council ; and zero-ablating the top-80 heads exported from the learned mask scores reverses this shift.

Explore similar work

Sentence-Level Contextual Entrainment in Large Language Models
Jun 23, 2026Yang Liu, Chenhui ChuLLM Interpretability
Kyoto University
Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
Apr 14, 2026Dikshant Kukreja, Kshitij Sah, Gautam Gupta +5Language Model Scaling Laws Knowledge Conflicts in Language Models
IIIT Delhi, India · Singapore Institute of Technology · NVIDIA +1
Task Context Shapes Component Roles in Language Models
Oct 9, 2026Sovesh Mohapatra, Dani S BassettLLM Interpretability Circuits in Language Models
University of Pennsylvania · Yale University

Jun 23, 2026J/K move · Enter open · S save

Sentence-Level Contextual Entrainment in Large Language Models

Yang Liu, Chenhui Chu

Kyoto University

LLM Interpretability

Open paper AskPDF arXiv

Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.

Abstract

Explore similar work

Sentence-Level Contextual Entrainment in Large Language Models

Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size

Task Context Shapes Component Roles in Language Models