Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

Amanda Fitch

cs.CLSep 25, 2026

Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

Authors: Amanda Fitch

Organizations: Google

Abstract

When a language model receives two conflicting documents as input, how does it decide which one to prioritize? Does it rely on how the sources are framed or the presentation order of the documents? We evaluated this behavior on Google's pre-trained Gemma 4-e4b model across a targeted behavioral suite (n = 13 items, 784 forward passes in short, single-turn contexts) using a completely counterbalanced experimental design. This setup allowed us to mathematically isolate the specific effects of source framing and reading position, while ensuring the model's natural vocabulary biases were canceled out. Across ten test conditions, we discovered the following: 1. Source framing heavily overpowers reading position. When directly competing, the semantic framing of a source (such as presenting it as an official guideline or a fresh update) had a significantly stronger impact on the model's final answer than the presentation order of the document. 2. The model favors the first document it reads, but this bias is highly variable. While the model consistently demonstrated a primacy effect (preferring the first document presented), the actual strength of this bias fluctuated by at least a factor of 5 based solely on the surface wording. 3. Overall structural repetition, not short copy-cues, drives positional bias. The model's preference for the first document is not a mechanical reaction to short, repetitive trigger phrases, such as "is [Answer]". However, the primacy effect does increase significantly when the two competing documents are structurally identical, using word-for-word verbatim templates. Introducing variation in the overall wording between the two sources reduces this positional bias.

Figures & tables

Figure 1: Comparison of source framing and positional bias in Gemma 4. When competing directly, the magnitude of the semantic framing effect outweighed the presentation order bias by a statistically significant 4-to-1 margin.

Cell	First block	Second block
$e_{1}$	Stale (Agile)	Recent (Waterfall)
$e_{2}$	Recent (Waterfall)	Stale (Agile)
$e_{3}$	Stale (Waterfall)	Recent (Agile)
$e_{4}$	Recent (Agile)	Stale (Waterfall)

Table 1: The four-way counterbalanced experimental design.

Dial type	Settings	What it changes
Header wording	neutral , framed	Whether the blocks are: neutral: labeled with an unbiased identifier blandly (e.g., “Document A”) framed: labeled as an official guideline (authority) or a fresh team update (recency). (e.g., authority: “Official Guideline:”) (e.g., recency: “Team Update:”)
Body wording	verbatim, mixed, loaded	The two block bodies are: verbatim: word-for-word identical except the answer word (e.g., “The required system is [Answer].”) mixed: reworded using difference sentence structures for each block. (e.g., Block 1: “The required system is [Answer],”) (e.g.,Block 2: “[Answer] is the recommended method.”) loaded: carries the framed meaning in the sentence itself. (e.g., “Official guidelines state the required system is [Answer].”)
Cue count	0, 1, 2	How many times the exact word pattern appears. (e.g. “is {e}”)

Table 2: Experimental variables used to construct the test conditions.

Test Condition	Header Wording	Body Wording	Cue Count
T1	neutral	verbatim	2
T1c	neutral	verbatim	0
T2	neutral	mixed	1
T2c	neutral	mixed	0
T3	framed	verbatim	2
T3c	framed	verbatim	0

Table 3: Configurations of the 10 qualifying test conditions.

Cell	Header	Body	Cues	Serial Position (half-effect, nats)	95% CI	Items Showing Primacy	90% CI clears -0.15
T1	neutral	verbatim	2	-0.485	[-0.611, -0.283]	12/13	$\checkmark$ [-0.592, -0.318]
T1c	neutral	verbatim	0	-0.884	[-1.074, -0.708]	13/13	$\checkmark$ [-1.044, -0.735]
T2	neutral	mixed	1	-0.377	[-0.496, -0.227]	12/13	$\checkmark$ [-0.478, -0.252]
T2c	neutral	mixed	0	-0.635	[-0.902, -0.452]	13/13	$\checkmark$ [-0.854, -0.478]
T3	framed	verbatim	2	-0.679	[-0.848, -0.520]	13/13	$\checkmark$ [-0.822, -0.543]
T3c	framed	verbatim	0	-0.569	[-0.739, -0.429]	13/13	$\checkmark$ [-0.713, -0.452]

Table 4: Positional bias across the 10 test conditions.

Contrast	Description	Filtered Dataset (13 items, primary)	Unfiltered Dataset (14 items)
A1	Cue removal, neutral header (T1c - T1)	-0.400 [-0.578, -0.237] *	-0.407 [-0.573, -0.251] *
A2	Cue removal, authority header (T3c - T3)	+0.110 [-0.081, +0.287]	+0.074 [-0.112, +0.257]
A3	Average of A1 and A2, descriptive only	-0.145 [-0.290, +0.010]	-0.166 [-0.306, -0.011] *
A4	Do A1 and A2 arms differ?	-0.509 [-0.687, -0.316] *	not computed $\ddagger$

Table 5: Effect of copy-cue removal on primacy (Contrasts A1–A4).

Quantity	Estimate	95% CI
Source framing (header + body together)	+1.054	[+0.907, +1.258] *
Serial position (half-effect)	-0.265	[-0.421, -0.137] *
Residual check (block-order null control)	+0.011	$\checkmark$ [-0.058, +0.087]
Token (base word preference)	+0.003	$\checkmark$ [-0.135, +0.165]

Table 6: Baseline forces and sanity checks in the cue-free condition (T5b).

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
D1	Framing vs. position margin, cue-free (T5b)	+0.774 [+0.585, +0.966] *	+0.806 [+0.616, +0.991] **
D2	Framing vs. position margin, with cues (T5)	+0.316 [+0.188, +0.434] *	+0.317 [+0.199, +0.428] *
D3	Effect of removing cues on framing (T5b - T5)	+0.539 [+0.330, +0.843] *	+0.583 [+0.366, +0.874] *

Table 7: Item-by-item comparison of source framing versus presentation order.

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
C1	Effect of authority header, mixed bodies (T4c - T2c)	+0.153 [-0.011, +0.352]	+0.123 [-0.033, +0.322]
C2	Effect of authority header, verbatim bodies (T3c - T1c)	+0.315 [+0.172, +0.420] *	+0.296 [+0.166, +0.403] *
C3	The interaction (C1 - C2)	-0.162 [-0.303, -0.034] *	-0.173 [-0.301, -0.048] *

Table 8: The interaction between authority headers and body wording variation.

ID	Purpose	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )	Status
A1	Neutral cue removal	-0.400 *	-0.407 *	Inferential
A2	Authority cue removal	+0.110	+0.074	Inferential (null)
A3	Pooled cue removal	-0.145	-0.166 *	Descriptive only
A4	Arm heterogeneity test	-0.509 *	not computed	Inferential; post-dates log
B1	Partial cue removal	+0.108	+0.110	Descriptive (confounded)
B2	Mixed-body cue removal	-0.258 *	-0.275 *	Descriptive (confounded)

Table 9: Comprehensive index of experimental contrasts.

Crossing	Block order	Entity-role assignment	Resulting prompt structure
1	Authority first	Agile = Stale, Waterfall = Recent	Official Guideline: Use Agile. Team Update: Use Waterfall.
2	Authority first	Waterfall = Stale, Agile = Recent	Official Guideline: Use Waterfall. Team Update: Use Agile.
3	Recent first	Agile = Stale, Waterfall = Recent	Team Update: Use Waterfall. Official Guideline: Use Agile.
4	Recent first	Waterfall = Stale, Agile = Recent	Team Update: Use Agile. Official Guideline: Use Waterfall.

Table 10: Four-way crossing configurations.

Nat value	CI	Odds (A to B)	Prob. of A	Prob. of B	Model’s preference
+2.0	[+1.80, +2.20]	7.39 to 1	88%	12%	Overwhelming preference for Option A.
+1.0	[+0.80, +1.20]	2.72 to 1	73%	27%	Strong preference for Option A.
0	[-0.20, +0.20]	1.00 to 1	50%	50%	Indifference. The model is guessing.
-1.0	[-1.20, -0.80]	1 to 2.72	27%	73%	Strong preference for Option B.
-2.0	[-2.20, -1.80]	1 to 7.39	12%	88%	Overwhelming preference for Option B.

Table 11: Conversion of log-odds (nats) to probabilities and relative odds.

Type	Role	Wording
Cued	Authority	Official guidelines state that for project planning, the required configuration is {e} .
Cue-Free	Authority	{e} is the required configuration for project planning, per official guidelines.
Cued	Recency	Update: The new updated specification for project planning is now {e} .
Cue-Free	Recency	{e} is the newly updated specification for project planning, effective now.

Table 12: Example prompt templates for cued and cue-free conditions.

Cell	Header	Body	Cues	Serial Position (half-effect, nats)	95% CI	Items Showing Primacy	90% CI clears -0.15
T1	neutral	verbatim	2	-0.485	[-0.611, -0.283]	12/13	$\checkmark$ [-0.592, -0.318]
T1c	neutral	verbatim	0	-0.884	[-1.074, -0.708]	13/13	$\checkmark$ [-1.044, -0.735]
T2	neutral	mixed	1	-0.377	[-0.496, -0.227]	12/13	$\checkmark$ [-0.478, -0.252]
T2c	neutral	mixed	0	-0.635	[-0.902, -0.452]	13/13	$\checkmark$ [-0.854, -0.478]
T3	framed	verbatim	2	-0.679	[-0.848, -0.520]	13/13	$\checkmark$ [-0.822, -0.543]
T3c	framed	verbatim	0	-0.569	[-0.739, -0.429]	13/13	$\checkmark$ [-0.713, -0.452]

Table 14

Contrast	Description	Filtered Dataset (13 items, primary)	Unfiltered Dataset (14 items)
A1	Cue removal, neutral header (T1c - T1)	-0.400 [-0.578, -0.237] *	-0.407 [-0.573, -0.251] *
A2	Cue removal, authority header (T3c - T3)	+0.110 [-0.081, +0.287]	+0.074 [-0.112, +0.257]
A3	Average of A1 and A2, descriptive only	-0.145 [-0.290, +0.010]	-0.166 [-0.306, -0.011] *
A4	Do A1 and A2 arms differ?	-0.509 [-0.687, -0.316] *	not computed $\ddagger$

Table 15

Test Condition	Header Wording	Body Wording	Cue Count
T5	framed	loaded	2
T5b	framed	loaded	0

Table 16

Quantity	Estimate	95% CI
Source framing (header + body together)	+1.054	[+0.907, +1.258] *
Serial position (half-effect)	-0.265	[-0.421, -0.137] *
Residual check (block-order null control)	+0.011	$\checkmark$ [-0.058, +0.087]
Token (base word preference)	+0.003	$\checkmark$ [-0.135, +0.165]

Table 17

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
D1	Framing vs. position margin, cue-free (T5b)	+0.774 [+0.585, +0.966] *	+0.806 [+0.616, +0.991] **
D2	Framing vs. position margin, with cues (T5)	+0.316 [+0.188, +0.434] *	+0.317 [+0.199, +0.428] *
D3	Effect of removing cues on framing (T5b - T5)	+0.539 [+0.330, +0.843] *	+0.583 [+0.366, +0.874] *

Table 18

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
C1	Effect of authority header, mixed bodies (T4c - T2c)	+0.153 [-0.011, +0.352]	+0.123 [-0.033, +0.322]
C2	Effect of authority header, verbatim bodies (T3c - T1c)	+0.315 [+0.172, +0.420] *	+0.296 [+0.166, +0.403] *
C3	The interaction (C1 - C2)	-0.162 [-0.303, -0.034] *	-0.173 [-0.301, -0.048] *

Table 19

Framing Effects in Language Models

Framing effects in language models study how wording, context, perspective, and presentation alter model judgments and outputs.

See all 51 papers

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
Sep 29, 2026Abhinav Rajeev Kumar, Paras ChopraLLM Sycophancy LLM Reliability
Lossfunk
Language Models Generalize to Human-like Word Order Preferences
Aug 5, 2026Amanda Popadich, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
Department of Linguistics, University of Washington
Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer
Oct 26, 2026Xinyi NiLLM Prompting Knowledge Conflicts in Language Models
Brandeis University

Sep 29, 2026J/K move · Enter open · S save

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Abhinav Rajeev Kumar, Paras Chopra

Lossfunk

LLM Sycophancy LLM Reliability Ownership Bias in Language Models

Open paper AskPDF arXiv

Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferring to the user. Yet the same models are far more compliant when a wrong answer is attributed to a verified source, which is how retrieval results, tool outputs, and grounded-search content often present information. We measure this gap across five open-weight families and three closed APIs. A single verified-source note endorsing a wrong answer flips 45-88% of baseline-correct responses in seven of eight models, and compliance rises with how authoritative the note sounds. Source deference and user agreement are not behaviorally interchangeable inside the model: on matched items with the same wrong answer, causal interventions can selectively suppress one without equally affecting the other. In three open-weight families, removing a fitted source direction lowers source compliance by 65-80 percentage points while removing a user or assistant direction has far smaller effects, and removing the user direction shows the reverse preference. A separately fitted intervention derived from source-versus-user cue activations moves compliance in both directions while leaving the prompt text unchanged. An authority direction fitted on trivia also transfers to PIQA and multi-turn SYCON dialogues without refitting, and removing it lowers wrong-source compliance by tens of percentage points in four of five families with no detected change in MMLU-Pro or GSM8K accuracy at our evaluation sizes. Source deference and user agreement therefore need separate evaluation.

Abstract

Framing Effects in Language Models

Explore similar work

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Language Models Generalize to Human-like Word Order Preferences

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer