Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size

Dikshant Kukreja; Kshitij Sah; Gautam Gupta; Avinash Anand; Rajiv Ratn Shah; Zhengkui Wang; Aik Beng Ng; Erik Cambria

Query	Context	Condition	Gold	Distractor
The capital of Germany is ___	The capital of France is Paris.	Related	Berlin	Paris
	Dolphins are mammals.	Irrelevant	Berlin	mammals
	Telescope.	Random	Berlin	Telescope
	The capital of Germany is Munich.	Counterfactual	Berlin	Munich
Sushi is a traditional dish from ___	Tacos are a traditional dish from Mexico.	Related	Japan	Mexico
	The sun rises in the east.	Irrelevant	Japan	east

Appendix

Table 2: Example queries with four context conditions. Each context introduces a distractor token that competes with the gold answer. Related contexts share the same relation type; Irrelevant contexts come from unrelated domains; Random contexts are single arbitrary tokens drawn from the Brown corpus ( Francis and Kučera, 1979 ) ; Counterfactual contexts assert false information using the query’s own relation structure. These examples illustrate the gradient of semantic coherence that underlies our experimental design.

		Distractor			Gold			Overall
Setting	Model	No	With	$\Delta$	No	With	$\Delta$	No	With	$\Delta$
Related	111M	3.07	10.13	7.06	4.68	6.03	1.35	1.62	$-$ 4.09	$-$ 5.71
	256M	3.08	9.88	6.81	5.23	7.29	2.06	2.15	$-$ 2.59	$-$ 4.74
	590M	3.37	9.49	6.12	6.01	8.31	2.30	2.64	$-$ 1.19	$-$ 3.83
	1.3B	3.73	8.93	5.20	6.72	9.93	3.21	2.99	1.00	$-$ 1.99
	2.7B	3.05	7.59	4.54	7.12	10.57	3.45	4.07	2.98	$-$ 1.09

Appendix

Table 3: Complete logit measurements across all Cerebras-GPT model sizes (111M–13B parameters). For each context condition, we report: logits without context (No), logits with context (With), and their difference (

\Delta

). Measurements are provided for distractor tokens, gold tokens, and overall preference (gold

-

distractor). Positive

\Delta_{\mathrm{dstr}}

indicates entrainment toward the distractor. The key patterns supporting our dual-mechanism hypothesis are visible: semantic contexts (Related, Counterfactual) show decreasing

\Delta_{\mathrm{dstr}}

with scale, while non-semantic contexts (Irrelevant, Random) show increasing

\Delta_{\mathrm{dstr}}

with scale.

A. Delta Metrics ( $\Delta$ = With Context $-$ No Context)
Metric	Setting	$R^{2}$	$b$ (Exponent)	95% CI	$p$ -value
$\Delta_{\mathrm{dstr}}$	Related	0.977	$-$ 0.135	[ $-$ 0.159, $-$ 0.111]	2.87e-05
	Irrelevant	0.879	$+$ 0.091	[ $+$ 0.052, $+$ 0.130]	1.80e-03
	Random	0.905	$+$ 0.217	[ $+$ 0.136, $+$ 0.298]	1.00e-03
	Counterfactual	0.926	$-$ 0.330	[ $-$ 0.438, $-$ 0.223]	5.21e-04
$\Delta_{\mathrm{overall}}$	Related	0.966	$-$ 0.514	[ $-$ 0.625, $-$ 0.403]	7.33e-05

Appendix

Table 4: Comprehensive scaling law statistics for Cerebras-GPT (111M–13B). All fits use

E(N)=a\cdot N^{b}

via linear regression in log-log space (

n=7

model sizes). Block A presents delta metrics—the primary focus of our analysis—showing the opposite-signed exponents between semantic contexts (negative

b

) and non-semantic contexts (positive

b

). Block B presents no-context baselines, confirming uniform scaling of intrinsic model capability (

b\approx+0.13

for gold tokens across all conditions). Block C presents with-context baselines. The consistency of baseline scaling validates that observed entrainment differences arise from context manipulation rather than dataset artifacts.

Setting	Model	No	With	$\Delta$	No	With	$\Delta$	No	With	$\Delta$
		Distractor			Gold			Overall
Related	410M	6.80	11.58	4.78	9.84	13.17	3.32	3.27	0.87	$-$ 2.41
	1B	7.59	11.89	4.30	11.06	14.43	3.37	3.41	2.01	$-$ 1.40
	1.4B	7.46	11.73	4.27	11.86	15.71	3.85	3.60	2.23	$-$ 1.37
	2.8B	7.08	11.29	4.21	11.60	14.51	2.90	3.97	2.79	$-$ 1.19
	6.9B	7.56	10.95	3.40	12.32	15.48	3.16	4.99	4.01	$-$ 0.98

Appendix

Table 5: Complete logit measurements across all Pythia model sizes (410M–12B parameters). Format mirrors Table 3 . The same directional patterns observed in Cerebras-GPT replicate here: semantic contexts (Related, Counterfactual) show decreasing

\Delta_{\mathrm{dstr}}

with scale, while non-semantic contexts (Irrelevant, Random) show increasing

\Delta_{\mathrm{dstr}}

. This cross-family replication supports the generality of our dual-mechanism interpretation.

A. Delta Metrics ( $\Delta$ = With Context $-$ No Context)
Metric	Setting	$R^{2}$	$b$ (Exponent)	95% CI	$p$ -value
$\Delta_{\mathrm{dstr}}$	Related	0.836	$-$ 0.089	[ $-$ 0.143, $-$ 0.034]	1.07e-02
	Irrelevant	0.938	$+$ 0.078	[ $+$ 0.050, $+$ 0.106]	1.46e-03
	Random	0.916	$+$ 0.156	[ $+$ 0.091, $+$ 0.222]	2.72e-03
	Counterfactual	0.998	$-$ 0.258	[ $-$ 0.273, $-$ 0.244]	9.90e-07
$\Delta_{\mathrm{overall}}$	Related	0.937	$-$ 0.306	[ $-$ 0.416, $-$ 0.196]	1.50e-03

Appendix

Table 6: Comprehensive scaling law statistics for Pythia (410M–12B). All fits use

E(N)=a\cdot N^{b}

via linear regression in log-log space (

n=6

model sizes). The sign split observed in Cerebras-GPT replicates exactly: semantic contexts (Counterfactual, Related) show negative exponents, while non-semantic contexts (Random, Irrelevant) show positive exponents. The Counterfactual condition achieves an exceptionally tight fit (

R^{2}=0.998

), providing strong evidence for precise power-law scaling of misinformation resistance.

Figure 4: Cerebras-GPT: Individual scaling fits for Related and Irrelevant contexts. (Left) Related: distractor entrainment decreases from 7.06 at 111M to 3.90 at 13B—a 1.8

\times

reduction. The negative exponent indicates that larger models show reduced susceptibility to topically related distractors, consistent with improved semantic filtering; the weaker slope relative to Counterfactual (

b=-0.33

) suggests that explicit contradiction triggers stronger resistance than mere topical similarity. (Right) Irrelevant: distractor entrainment increases from 3.02 at 111M to 4.61 at 13B—a 1.5

\times

increase. The positive exponent indicates that larger models show slightly increased copying of semantically unrelated tokens; the weaker slope relative to Random (

b=+0.22

) is consistent with the hypothesis that even minimal semantic scaffolding (grammatically coherent sentences) partially engages the filtering mechanism. Shaded regions show 95% confidence intervals.

Figure 5: Pythia: Scaling of distractor entrainment across model sizes. Combined visualization showing opposite scaling trends for semantic vs. non-semantic contexts. (Left) Counterfactual context shows negative scaling (

b=-0.26

,

R^{2}=0.998

), with entrainment decreasing from 4.85 at 410M to 2.06 at 12B—a 2.4

\times

reduction indicating improved resistance to misinformation at scale. (Right) Random context shows positive scaling (

b=+0.16

,

R^{2}=0.92

), with entrainment increasing from 1.68 at 410M to 2.78 at 12B—a 1.7

\times

increase indicating enhanced mechanical token copying at scale. This pattern exactly replicates the Cerebras-GPT findings (Figure 2 in main text), with the same sign split between semantic and non-semantic contexts, confirming that the dual-mechanism interpretation generalizes across independently trained model families.

Figure 6: Pythia: Individual scaling fits for Related and Irrelevant contexts. (Left) Related: distractor entrainment decreases from 4.78 at 410M to 3.69 at 12B. The negative exponent matches the Cerebras-GPT pattern (Figure 4(a) ), confirming cross-family generalization of improved semantic filtering at scale; the lower

R^{2}

compared to Counterfactual (

R^{2}=0.998

) reflects the weaker scaling signal in Related contexts. (Right) Irrelevant: distractor entrainment increases from 2.09 at 410M to 2.72 at 12B. The positive exponent is consistent with the Cerebras-GPT findings (Figure 4(b) ), demonstrating that mechanical copying generalizes across model families; the higher

R^{2}

in Pythia suggests more regular scaling behavior for this condition. Shaded regions show 95% confidence intervals.

Figure 7: Cerebras-GPT: Convergence vs. divergence for Irrelevant and Counterfactual contexts. Joint scaling of gold entrainment (

\Delta_{\mathrm{gold}}

, blue) and distractor entrainment (

\Delta_{\mathrm{dstr}}

, red). (Left) Irrelevant: gold entrainment is essentially flat (

b_{\mathrm{gold}}\approx 0

, ranging from 0.05 to 0.17) while distractor entrainment increases (

b_{\mathrm{dstr}}=+0.09

, from 3.02 to 4.61); the gap widens from 2.97 at 111M to 4.44 at 13B (1.5

\times

increase). This divergent pattern indicates that models become relatively more distracted by irrelevant context at scale—because Irrelevant contexts lack semantic content, the filtering mechanism does not engage, leaving only mechanical copying to scale upward. (Right) Counterfactual: distractor entrainment drops sharply (

b_{\mathrm{dstr}}=-0.33

, from 9.69 to 2.30) while gold entrainment remains relatively stable (0.79 to 3.44); the gap narrows from 6.50 at 111M to 1.07 at 13B (6.1

\times

reduction). This convergent pattern demonstrates effective misinformation resistance at scale: larger models not only suppress false claims but maintain or boost the correct answer.

Figure 8: Pythia: Convergence vs. Divergence Across Context Types. Combined visualization showing opposite convergence behaviors for semantic vs. non-semantic contexts. (Left) Related context shows convergent behavior: the gold-distractor gap narrows with scale as distractor entrainment decreases (

b_{\mathrm{dstr}}=-0.09

) while gold entrainment remains stable. The gap reduces from 2.41 at 410M to 0.74 at 12B—a 3.3

\times

improvement in discrimination. (Right) Random context shows divergent behavior: the gap widens with scale as distractor entrainment increases (

b_{\mathrm{dstr}}=+0.16

) while gold entrainment stays flat. The gap increases from 1.44 at 410M to 2.13 at 12B—a 1.5

\times

degradation. This pattern exactly replicates the Cerebras-GPT findings (Figure 3 in main text), confirming that the convergent-divergent split is a general property of Transformer scaling, not a family-specific artifact. The replication across independently trained model families provides strong evidence for the dual-mechanism interpretation. Figure 10 (b) additionally presents the Pythia Irrelevant divergence plot in isolation.

Figure 9: Cerebras-GPT: Log-log verification and entrainment heatmap. (Left) Distractor entrainment (

\Delta_{\mathrm{dstr}}

) plotted against model size (

N

) in log-log space for all four context conditions. Power-law relationships

\Delta_{\mathrm{dstr}}\propto N^{b}

appear as straight lines with slope

b

. Semantic contexts (Counterfactual, Related) show negative slopes (downward trends), while non-semantic contexts (Irrelevant, Random) show positive slopes (upward trends). The linearity across nearly two orders of magnitude in model size (111M–13B) confirms that power laws accurately describe entrainment scaling, and the parallel structure of lines within each group suggests a common underlying mechanism with condition-specific magnitudes. (Right) Matrix visualization of

\Delta_{\mathrm{dstr}}

across all combinations of context condition (rows) and model size (columns, 111M–13B). Color intensity indicates entrainment magnitude. The heatmap reveals two distinct gradients: semantic contexts decrease from left to right, while non-semantic contexts increase. The Counterfactual row shows the steepest gradient (strongest negative scaling), while the Random row shows clear brightening with scale (strongest positive scaling).

Figure 10: Pythia: Scaling exponents and Irrelevant-context divergence. (Left) Estimated power-law exponents (

b

) for distractor entrainment (

\Delta_{\mathrm{dstr}}

) across the four context conditions, with 95% confidence intervals shown as error bars. Semantic contexts (Counterfactual:

b=-0.26

; Related:

b=-0.09

) show negative exponents, while non-semantic contexts (Random:

b=+0.16

; Irrelevant:

b=+0.08

) show positive exponents. The confidence intervals for semantic and non-semantic groups do not overlap, establishing statistical separation. This pattern exactly replicates the Cerebras-GPT findings (Figure 1 in main text), with the same ordering of exponent magnitudes:

|b_{\mathrm{CF}}|>|b_{\mathrm{Rel}}|

and

|b_{\mathrm{Rnd}}|>|b_{\mathrm{Irr}}|

. (Right) Joint scaling of gold entrainment (

\Delta_{\mathrm{gold}}

, blue) and distractor entrainment (

\Delta_{\mathrm{dstr}}

, red) for the Irrelevant context condition. Similar to Cerebras-GPT (Figure 7(a) ), gold entrainment remains essentially flat across scale (0.29 to 0.38) while distractor entrainment increases (

b_{\mathrm{dstr}}=+0.08

, from 2.09 to 2.72); the gap widens from 1.89 at 410M to 2.22 at 12B. This divergent pattern replicates across model families, confirming that the mechanical copying mechanism scales independently of semantic content.

Figure 11: Pythia: Log-log verification and entrainment heatmap. (Left) Distractor entrainment (

\Delta_{\mathrm{dstr}}

) plotted against model size (

N

) in log-log space for all four context conditions. As with Cerebras-GPT (Figure 9(a) ), power-law relationships appear as straight lines. The Counterfactual condition shows particularly tight linearity (

R^{2}=0.998

), providing strong evidence that misinformation resistance scales as a precise power law. The semantic/non-semantic split in slopes replicates across model families, confirming that

\Delta_{\mathrm{dstr}}\propto N^{b}

with opposite-signed

b

is a general scaling pattern. (Right) Matrix visualization of

\Delta_{\mathrm{dstr}}

across all combinations of context condition (rows) and model size (columns, 410M–12B). The same divergent gradient pattern observed in Cerebras-GPT (Figure 9(b) ) emerges here: semantic contexts (Counterfactual, Related) darken from left to right (decreasing entrainment), while non-semantic contexts (Irrelevant, Random) brighten (increasing entrainment). The overall lower intensity compared to Cerebras-GPT reflects Pythia’s generally smaller entrainment magnitudes, though the directional trends are identical.

Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size

Abstract

Figures & tables

Appendix

Explore similar work

Sentence-Level Contextual Entrainment in Large Language Models

From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

Memory vs. Context? Influential Factors of Factual Recall in Language Models