Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

Amanda Fitch

cs.CLSep 25, 2026

Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

Authors: Amanda Fitch

Organizations: Google

Abstract

When a language model receives two conflicting documents as input, how does it decide which one to prioritize? Does it rely on how the sources are framed or the presentation order of the documents? We evaluated this behavior on Google's pre-trained Gemma 4-e4b model across a targeted behavioral suite (n = 13 items, 784 forward passes in short, single-turn contexts) using a completely counterbalanced experimental design. This setup allowed us to mathematically isolate the specific effects of source framing and reading position, while ensuring the model's natural vocabulary biases were canceled out. Across ten test conditions, we discovered the following: 1. Source framing heavily overpowers reading position. When directly competing, the semantic framing of a source (such as presenting it as an official guideline or a fresh update) had a significantly stronger impact on the model's final answer than the presentation order of the document. 2. The model favors the first document it reads, but this bias is highly variable. While the model consistently demonstrated a primacy effect (preferring the first document presented), the actual strength of this bias fluctuated by at least a factor of 5 based solely on the surface wording. 3. Overall structural repetition, not short copy-cues, drives positional bias. The model's preference for the first document is not a mechanical reaction to short, repetitive trigger phrases, such as "is [Answer]". However, the primacy effect does increase significantly when the two competing documents are structurally identical, using word-for-word verbatim templates. Introducing variation in the overall wording between the two sources reduces this positional bias.

Figures & tables

Figure 1: Comparison of source framing and positional bias in Gemma 4. When competing directly, the magnitude of the semantic framing effect outweighed the presentation order bias by a statistically significant 4-to-1 margin.

Cell	First block	Second block
$e_{1}$	Stale (Agile)	Recent (Waterfall)
$e_{2}$	Recent (Waterfall)	Stale (Agile)
$e_{3}$	Stale (Waterfall)	Recent (Agile)
$e_{4}$	Recent (Agile)	Stale (Waterfall)

Table 1: The four-way counterbalanced experimental design.

Dial type	Settings	What it changes
Header wording	neutral , framed	Whether the blocks are: neutral: labeled with an unbiased identifier blandly (e.g., “Document A”) framed: labeled as an official guideline (authority) or a fresh team update (recency). (e.g., authority: “Official Guideline:”) (e.g., recency: “Team Update:”)
Body wording	verbatim, mixed, loaded	The two block bodies are: verbatim: word-for-word identical except the answer word (e.g., “The required system is [Answer].”) mixed: reworded using difference sentence structures for each block. (e.g., Block 1: “The required system is [Answer],”) (e.g.,Block 2: “[Answer] is the recommended method.”) loaded: carries the framed meaning in the sentence itself. (e.g., “Official guidelines state the required system is [Answer].”)
Cue count	0, 1, 2	How many times the exact word pattern appears. (e.g. “is {e}”)

Table 2: Experimental variables used to construct the test conditions.

Test Condition	Header Wording	Body Wording	Cue Count
T1	neutral	verbatim	2
T1c	neutral	verbatim	0
T2	neutral	mixed	1
T2c	neutral	mixed	0
T3	framed	verbatim	2
T3c	framed	verbatim	0

Table 3: Configurations of the 10 qualifying test conditions.

Cell	Header	Body	Cues	Serial Position (half-effect, nats)	95% CI	Items Showing Primacy	90% CI clears -0.15
T1	neutral	verbatim	2	-0.485	[-0.611, -0.283]	12/13	$\checkmark$ [-0.592, -0.318]
T1c	neutral	verbatim	0	-0.884	[-1.074, -0.708]	13/13	$\checkmark$ [-1.044, -0.735]
T2	neutral	mixed	1	-0.377	[-0.496, -0.227]	12/13	$\checkmark$ [-0.478, -0.252]
T2c	neutral	mixed	0	-0.635	[-0.902, -0.452]	13/13	$\checkmark$ [-0.854, -0.478]
T3	framed	verbatim	2	-0.679	[-0.848, -0.520]	13/13	$\checkmark$ [-0.822, -0.543]
T3c	framed	verbatim	0	-0.569	[-0.739, -0.429]	13/13	$\checkmark$ [-0.713, -0.452]

Table 4: Positional bias across the 10 test conditions.

Contrast	Description	Filtered Dataset (13 items, primary)	Unfiltered Dataset (14 items)
A1	Cue removal, neutral header (T1c - T1)	-0.400 [-0.578, -0.237] *	-0.407 [-0.573, -0.251] *
A2	Cue removal, authority header (T3c - T3)	+0.110 [-0.081, +0.287]	+0.074 [-0.112, +0.257]
A3	Average of A1 and A2, descriptive only	-0.145 [-0.290, +0.010]	-0.166 [-0.306, -0.011] *
A4	Do A1 and A2 arms differ?	-0.509 [-0.687, -0.316] *	not computed $\ddagger$

Table 5: Effect of copy-cue removal on primacy (Contrasts A1–A4).

Quantity	Estimate	95% CI
Source framing (header + body together)	+1.054	[+0.907, +1.258] *
Serial position (half-effect)	-0.265	[-0.421, -0.137] *
Residual check (block-order null control)	+0.011	$\checkmark$ [-0.058, +0.087]
Token (base word preference)	+0.003	$\checkmark$ [-0.135, +0.165]

Table 6: Baseline forces and sanity checks in the cue-free condition (T5b).

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
D1	Framing vs. position margin, cue-free (T5b)	+0.774 [+0.585, +0.966] *	+0.806 [+0.616, +0.991] **
D2	Framing vs. position margin, with cues (T5)	+0.316 [+0.188, +0.434] *	+0.317 [+0.199, +0.428] *
D3	Effect of removing cues on framing (T5b - T5)	+0.539 [+0.330, +0.843] *	+0.583 [+0.366, +0.874] *

Table 7: Item-by-item comparison of source framing versus presentation order.

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
C1	Effect of authority header, mixed bodies (T4c - T2c)	+0.153 [-0.011, +0.352]	+0.123 [-0.033, +0.322]
C2	Effect of authority header, verbatim bodies (T3c - T1c)	+0.315 [+0.172, +0.420] *	+0.296 [+0.166, +0.403] *
C3	The interaction (C1 - C2)	-0.162 [-0.303, -0.034] *	-0.173 [-0.301, -0.048] *

Table 8: The interaction between authority headers and body wording variation.

ID	Purpose	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )	Status
A1	Neutral cue removal	-0.400 *	-0.407 *	Inferential
A2	Authority cue removal	+0.110	+0.074	Inferential (null)
A3	Pooled cue removal	-0.145	-0.166 *	Descriptive only
A4	Arm heterogeneity test	-0.509 *	not computed	Inferential; post-dates log
B1	Partial cue removal	+0.108	+0.110	Descriptive (confounded)
B2	Mixed-body cue removal	-0.258 *	-0.275 *	Descriptive (confounded)

Table 9: Comprehensive index of experimental contrasts.

Crossing	Block order	Entity-role assignment	Resulting prompt structure
1	Authority first	Agile = Stale, Waterfall = Recent	Official Guideline: Use Agile. Team Update: Use Waterfall.
2	Authority first	Waterfall = Stale, Agile = Recent	Official Guideline: Use Waterfall. Team Update: Use Agile.
3	Recent first	Agile = Stale, Waterfall = Recent	Team Update: Use Waterfall. Official Guideline: Use Agile.
4	Recent first	Waterfall = Stale, Agile = Recent	Team Update: Use Agile. Official Guideline: Use Waterfall.

Table 10: Four-way crossing configurations.

Nat value	CI	Odds (A to B)	Prob. of A	Prob. of B	Model’s preference
+2.0	[+1.80, +2.20]	7.39 to 1	88%	12%	Overwhelming preference for Option A.
+1.0	[+0.80, +1.20]	2.72 to 1	73%	27%	Strong preference for Option A.
0	[-0.20, +0.20]	1.00 to 1	50%	50%	Indifference. The model is guessing.
-1.0	[-1.20, -0.80]	1 to 2.72	27%	73%	Strong preference for Option B.
-2.0	[-2.20, -1.80]	1 to 7.39	12%	88%	Overwhelming preference for Option B.

Table 11: Conversion of log-odds (nats) to probabilities and relative odds.

Type	Role	Wording
Cued	Authority	Official guidelines state that for project planning, the required configuration is {e} .
Cue-Free	Authority	{e} is the required configuration for project planning, per official guidelines.
Cued	Recency	Update: The new updated specification for project planning is now {e} .
Cue-Free	Recency	{e} is the newly updated specification for project planning, effective now.

Table 12: Example prompt templates for cued and cue-free conditions.

Cell	Header	Body	Cues	Serial Position (half-effect, nats)	95% CI	Items Showing Primacy	90% CI clears -0.15
T1	neutral	verbatim	2	-0.485	[-0.611, -0.283]	12/13	$\checkmark$ [-0.592, -0.318]
T1c	neutral	verbatim	0	-0.884	[-1.074, -0.708]	13/13	$\checkmark$ [-1.044, -0.735]
T2	neutral	mixed	1	-0.377	[-0.496, -0.227]	12/13	$\checkmark$ [-0.478, -0.252]
T2c	neutral	mixed	0	-0.635	[-0.902, -0.452]	13/13	$\checkmark$ [-0.854, -0.478]
T3	framed	verbatim	2	-0.679	[-0.848, -0.520]	13/13	$\checkmark$ [-0.822, -0.543]
T3c	framed	verbatim	0	-0.569	[-0.739, -0.429]	13/13	$\checkmark$ [-0.713, -0.452]

Table 14

Contrast	Description	Filtered Dataset (13 items, primary)	Unfiltered Dataset (14 items)
A1	Cue removal, neutral header (T1c - T1)	-0.400 [-0.578, -0.237] *	-0.407 [-0.573, -0.251] *
A2	Cue removal, authority header (T3c - T3)	+0.110 [-0.081, +0.287]	+0.074 [-0.112, +0.257]
A3	Average of A1 and A2, descriptive only	-0.145 [-0.290, +0.010]	-0.166 [-0.306, -0.011] *
A4	Do A1 and A2 arms differ?	-0.509 [-0.687, -0.316] *	not computed $\ddagger$

Table 15

Test Condition	Header Wording	Body Wording	Cue Count
T5	framed	loaded	2
T5b	framed	loaded	0

Table 16

Quantity	Estimate	95% CI
Source framing (header + body together)	+1.054	[+0.907, +1.258] *
Serial position (half-effect)	-0.265	[-0.421, -0.137] *
Residual check (block-order null control)	+0.011	$\checkmark$ [-0.058, +0.087]
Token (base word preference)	+0.003	$\checkmark$ [-0.135, +0.165]

Table 17

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
D1	Framing vs. position margin, cue-free (T5b)	+0.774 [+0.585, +0.966] *	+0.806 [+0.616, +0.991] **
D2	Framing vs. position margin, with cues (T5)	+0.316 [+0.188, +0.434] *	+0.317 [+0.199, +0.428] *
D3	Effect of removing cues on framing (T5b - T5)	+0.539 [+0.330, +0.843] *	+0.583 [+0.366, +0.874] *

Table 18

Contrast	Description	Filtered Dataset ( $n=13$ )	Unfiltered Dataset ( $n=14$ )
C1	Effect of authority header, mixed bodies (T4c - T2c)	+0.153 [-0.011, +0.352]	+0.123 [-0.033, +0.322]
C2	Effect of authority header, verbatim bodies (T3c - T1c)	+0.315 [+0.172, +0.420] *	+0.296 [+0.166, +0.403] *
C3	The interaction (C1 - C2)	-0.162 [-0.303, -0.034] *	-0.173 [-0.301, -0.048] *

Table 19

Framing Effects in Language Models

Framing effects in language models study how wording, context, perspective, and presentation alter model judgments and outputs.

See all 51 papers

Sep 29, 2026

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferring to the user. Yet the same models are far more compliant when a wrong answer is attributed to a verified source, which is how retrieval results, tool outputs, and grounded-search content often present information. We measure this gap across five open-weight families and three closed APIs. A single verified-source note endorsing a wrong answer flips 45-88% of baseline-correct responses in seven of eight models, and compliance rises with how authoritative the note sounds. Source deference and user agreement are not behaviorally interchangeable inside the model: on matched items with the same wrong answer, causal interventions can selectively suppress one without equally affecting the other. In three open-weight families, removing a fitted source direction lowers source compliance by 65-80 percentage points while removing a user or assistant direction has far smaller effects, and removing the user direction shows the reverse preference. A separately fitted intervention derived from source-versus-user cue activations moves compliance in both directions while leaving the prompt text unchanged. An authority direction fitted on trivia also transfers to PIQA and multi-turn SYCON dialogues without refitting, and removing it lowers wrong-source compliance by tens of percentage points in four of five families with no detected change in MMLU-Pro or GSM8K accuracy at our evaluation sizes. Source deference and user agreement therefore need separate evaluation.

Abhinav Rajeev Kumar, Paras Chopra

Lossfunk

LLM Sycophancy LLM Reliability Ownership Bias in Language Models

Ask PDF

Aug 5, 2026

Language Models Generalize to Human-like Word Order Preferences

A central question in language acquisition is whether linguistic biases can emerge from general learning mechanisms operating over underdetermined input. Artificial Language Learning (ALL) studies have shown that human learners reliably generalize beyond the evidence provided, including by preferring scope-homomorphic noun phrase modifier orders. In this work, we investigate whether language models exhibit the same bias under similar conditions. We create a controlled learning environment in which models are trained on a corpus where all noun phrases containing multiple modifiers have been removed, eliminating direct evidence about modifier ordering, and are then evaluated on multiple modifier sentences. Across three model sizes, we find that they consistently prefer scope-homomorphic orders despite never observing them during training. These preferences vary in strength by modifier type. To investigate the source of these preferences, we examine noun-modifier association strength using pointwise mutual information (PMI). While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences. These findings demonstrate that LMs can recover human-like linguistic generalizations from impoverished input and provide a controlled framework for investigating the mechanisms underlying such biases.

Amanda Popadich, Shane Steinert-Threlkeld

Department of Linguistics, University of Washington

Syntactic Generalization in Language Models Language Modeling Linguistic Bias in Language Models

Ask PDF

Oct 26, 2026

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer

Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We ask what it takes to talk a model out of an answer it gets right on its own, and separate three factors that earlier studies vary together: where the advice sits in the prompt, who is said to have written it, and what the model has already written. On Qwen3.5-27B and four-option MMLU-Pro items, with the rest of the advice held fixed in each comparison, we find three things. First, position matters far more than the API channel, in opposite directions for a bare claim and an argument: a bare "the answer is X" does most harm just before the model answers, while the same claim with a false rationale does most harm before the question, where it raises error on items the model otherwise almost never misses from 1% to 67%; moving the claim between the system message and the user message at the same position changes error by a few points. Second, attributing the note to an authority rather than to another participant, and phrasing it assertively, each raise error by 10-27 points, while the two authorities we tested differ by under two. Third, replaying a correct worked answer before the advice protects it whether the text is shown as the model's own or as someone else's, and the same replay keeps wrong answers from being corrected by a bare correct note. The position result replicates under an independently written prompt. Code, items and all result tables are released at https://github.com/coolkillercat/safety.

Xinyi Ni

Brandeis University

LLM Prompting Knowledge Conflicts in Language Models LLM Reliability

Ask PDF

Abstract

Framing Effects in Language Models

Explore similar work

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Language Models Generalize to Human-like Word Order Preferences

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer