Lexical Redefinition Reveals Context-Sensitive Instruction Following

Roksana Goworek; Haim Dubossarsky

cs.CLOct 12, 2026

Lexical Redefinition Reveals Context-Sensitive Instruction Following

Authors: Roksana Goworek, Haim Dubossarsky

Organizations: Queen Mary University of London · University of Cambridge

Abstract

How do language models reconcile conflicting contextual evidence within a prompt? We study this question using lexical redefinition, where an explicit instruction specifies how a word should be interpreted and a subsequent sentence provides additional evidence through its usage. Across 870 word pairs, we find that lexical redefinitions are not applied as fixed rules. Instead, the behaviour depends systematically on the relationship between the explicit instruction and the intervening sentence: the contextualised usage can preserve, weaken, or completely reverse the instruction. These results suggest that the interpretation of an explicit instruction remains sensitive to subsequent contextual evidence. Lexical redefinition therefore provides a controlled behavioural paradigm for studying how language models integrate multiple sources of information within a prompt.

Figures & tables

Figure 1: Token probabilities for sentence-completion queries under redefinition prompts and the original prompt without redefinition (" $\varnothing$ "). Sentences and icons are illustrative, results are aggregated over 870 word pairs. Error bars show SD.

Sentence
“The mouse eats crumbs” $\therefore implies$	Compatible uses in its ordinary sense, but the redefinition affects only contexts $\rightarrow$ redefinition unaffected or slightly weakened	Competes uses in its ordinary sense, which was redefined to be denoted by $\rightarrow$ adopted instead
“The letter eats crumbs” $\therefore implies$	Contradicts shows an opposite mapping to the one in the instruction $\rightarrow$ adopted instead	Reinforces shows an example of the redefinition $\rightarrow$ redefinition effect is unchanged

Table 1: Analysis of the four sentence–redefinition combinations. Each cell names the relationship between the sentence and the redefinition instruction, and the observed effect on the completions.

Figure 2: Effect of the intervening sentence on completion probabilities ( $\Delta P$ , Eq. 2 ). Blue/orange bars show $\Delta P$ for mouse / letter . Percentages inside boxes indicate sign agreement with the mean across word pairs (Eq. 3 ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Nouns	Verbs	Adjectives
truth	run	hot
chair	know	cold
dog	break	big
flower	give	small
bank	think	good
river	find	bad

Appendix

Table 2: The 30-word stimulus set by part of speech.

Wording	Prompt (X, Y stand for the word pair)
baseline	In the following sentences, the word ‘X’ should be interpreted as if it were the word ‘Y’.
scope_sentence_only	For the next sentence only, ‘X’ should be interpreted as if it were ‘Y’.
scope_global	For the rest of this conversation, ‘X’ should be interpreted as if it were ‘Y’.
denotational	Whenever ‘X’ appears, read it as ‘Y’.
comprehension_only	Treat ‘X’ as having the meaning of ‘Y’ when reading. Do not change how you produce these words; only how you interpret them.

Appendix

Table 3: The baseline redefinition prompt and four alternative formulations.

Figure 3: Qwen2.5-14B-Instruct: raw next-token probabilities. Rows: context sentence (icon) $\times$ query, plus two no-sentence rows; columns: redefinition condition (icon). Bars: P(mouse) $=w_{1}$ , P(letter) $=w_{2}$ ; $\pm 1$ SD over pairs.

		$\varnothing$
context	query
		.10 [1pt] $\pm$ .14	.00 [1pt] $\pm$ .00	.07 [1pt] $\pm$ .20	.01 [1pt] $\pm$ .02	.34 [1pt] $\pm$ .19	-.28 [1pt] $\pm$ .26
		.00 [1pt] $\pm$ .01	.07 [1pt] $\pm$ .10	.05 [1pt] $\pm$ .22	.02 [1pt] $\pm$ .02	.30 [1pt] $\pm$ .23	-.29 [1pt] $\pm$ .25
		.00 [1pt] $\pm$ .00	.10 [1pt] $\pm$ .19	-.36 [1pt] $\pm$ .28	.50 [1pt] $\pm$ .21	.00 [1pt] $\pm$ .01	.16 [1pt] $\pm$ .23
		.04 [1pt] $\pm$ .08	.01 [1pt] $\pm$ .02	-.44 [1pt] $\pm$ .28	.52 [1pt] $\pm$ .27	.00 [1pt] $\pm$ .02	.15 [1pt] $\pm$ .28

Appendix

Table 4: No-sentence-normalised sentence effect

\Delta P

(

P_{\text{with sentence}}-P_{\text{no sentence}}

) for Qwen2.5-14B-Instruct under the scope_global wording (8-word variant subset). Rows: context sentence

\times

query word; columns: redefinition direction

\times

queried word. Each cell: mean

\Delta P

over the 56 pairs;

\pm

line is the SD over pairs.

		$\varnothing$
context	query
		.10 [1pt] $\pm$ .14	.00 [1pt] $\pm$ .00	-.15 [1pt] $\pm$ .16	.02 [1pt] $\pm$ .03	.20 [1pt] $\pm$ .15	-.21 [1pt] $\pm$ .21
		.00 [1pt] $\pm$ .01	.07 [1pt] $\pm$ .10	-.08 [1pt] $\pm$ .18	.02 [1pt] $\pm$ .03	.11 [1pt] $\pm$ .15	-.19 [1pt] $\pm$ .22
		.00 [1pt] $\pm$ .00	.10 [1pt] $\pm$ .19	-.30 [1pt] $\pm$ .23	.29 [1pt] $\pm$ .18	.00 [1pt] $\pm$ .01	-.10 [1pt] $\pm$ .22
		.04 [1pt] $\pm$ .08	.01 [1pt] $\pm$ .02	-.35 [1pt] $\pm$ .21	.21 [1pt] $\pm$ .17	.02 [1pt] $\pm$ .02	-.14 [1pt] $\pm$ .20

Appendix

Table 5: Prompt-variation results for Qwen2.5-14B-Instruct ( scope_sentence_only wording).

		$\varnothing$
context	query
		.10 [1pt] $\pm$ .14	.00 [1pt] $\pm$ .00	-.02 [1pt] $\pm$ .24	-.09 [1pt] $\pm$ .18	.22 [1pt] $\pm$ .23	-.13 [1pt] $\pm$ .18
		.00 [1pt] $\pm$ .01	.07 [1pt] $\pm$ .10	-.04 [1pt] $\pm$ .13	-.01 [1pt] $\pm$ .21	.25 [1pt] $\pm$ .16	-.20 [1pt] $\pm$ .23
		.00 [1pt] $\pm$ .00	.10 [1pt] $\pm$ .19	-.15 [1pt] $\pm$ .19	.24 [1pt] $\pm$ .26	-.13 [1pt] $\pm$ .17	.11 [1pt] $\pm$ .27
		.04 [1pt] $\pm$ .08	.01 [1pt] $\pm$ .02	-.24 [1pt] $\pm$ .24	.26 [1pt] $\pm$ .17	-.07 [1pt] $\pm$ .21	.05 [1pt] $\pm$ .16

Appendix

Table 6: Prompt-variation results for Qwen2.5-14B-Instruct ( denotational wording).

		$\varnothing$
context	query
		.10 [1pt] $\pm$ .14	.00 [1pt] $\pm$ .00	.01 [1pt] $\pm$ .24	.00 [1pt] $\pm$ .01	.12 [1pt] $\pm$ .10	-.17 [1pt] $\pm$ .16
		.00 [1pt] $\pm$ .01	.07 [1pt] $\pm$ .10	.01 [1pt] $\pm$ .23	.00 [1pt] $\pm$ .01	.13 [1pt] $\pm$ .12	-.12 [1pt] $\pm$ .22
		.00 [1pt] $\pm$ .00	.10 [1pt] $\pm$ .19	-.33 [1pt] $\pm$ .24	.36 [1pt] $\pm$ .16	.00 [1pt] $\pm$ .01	.08 [1pt] $\pm$ .29
		.04 [1pt] $\pm$ .08	.01 [1pt] $\pm$ .02	-.38 [1pt] $\pm$ .19	.35 [1pt] $\pm$ .18	.00 [1pt] $\pm$ .01	.00 [1pt] $\pm$ .26

Appendix

Table 7: Prompt-variation results for Qwen2.5-14B-Instruct ( comprehension_only wording).

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 419 papers

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
May 19, 2026Carolina Camassa, Derek ShillerLLM Evaluation Language Model Self-Assessment
Future Impact Group · Rethink Priorities
Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override
May 25, 2026Han-yu WangKnowledge Conflicts in Language Models LLM Interpretability
The University of Hong Kong
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
Jul 21, 2026Netanel EliavLLM Evaluation Language Model Generation Evaluation
Machine Human Intelligence Lab (MHIL)

May 19, 2026J/K move · Enter open · S save

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

Carolina Camassa, Derek Shiller

Future Impact Group · Rethink Priorities

LLM Evaluation Language Model Self-Assessment Instruction Following In-Context Learning Language Model Robustness

Open paper AskPDF arXiv

Language models are trained to follow instructions, but they are also powerful pattern completers. What happens when these two objectives conflict? We construct conversations in which a user instruction to behave in a target way T (e.g., always output a specific token, answer in a particular language, or adopt a persona) is opposed by N hardcoded assistant turns demonstrating a competing pattern P. We then measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns. Average instruction-following rates range from 1% to 99% across models, largely uncorrelated with standard capability benchmarks. The transition from instruction-following to pattern-following is universal but highly model-dependent. Robustness is modulated both by instruction content, with models resisting induction longer when instructions align with their trained value priors, and by output format, with diverse multi-token responses proving substantially more resistant than single-token outputs. Chain-of-thought reasoning improves robustness but does not eliminate susceptibility, and can produce dissociation between correct deliberation and incorrect output. When asked to predict their behavior in this setting, models achieve 83.5% accuracy on average but systematically underestimate their own resistance to induction pressure. These results suggest that instruction-following remains brittle under induction pressure even for otherwise capable models, and that output diversity, rather than semantic engagement with the input, is the primary factor predicting robustness.

Abstract

LLM Prompting

Explore similar work

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override

Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models