Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation

Dongxu Wu; Sara Court; Micha Elsner

cs.CLNov 6, 2026

Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation

Authors: Dongxu Wu, Sara Court, Micha Elsner

Organizations: The Ohio State University

Abstract

External lexical resources can help large language models (LLMs) translate texts in low-resource languages and specialized domains, but these resources may conflict with models' prior lexical knowledge. To isolate this conflict, we construct a controlled Mandarin-to-English translation task in which familiar source words are assigned new meanings within and across semantic categories. We treat these prompt-supplied meanings as contextual knowledge (CK) and systematically evaluate the effects of word frequency, sentence context, and prompt framing on the translated outputs. Across nine models, CK reliance is greater for less frequent source words and in sentence contexts that support the supplied meanings. Prompt framing can reduce CK reliance, but seven of nine models favor CK even when explicitly instructed to ignore the supplied meanings. Within model families, larger models are more likely than smaller models to utilize CK in their outputs. Three follow-up experiments further show that modal weakening lowers CK reliance, the effect of repetition depends on prompt layout, and the effect of reasoning varies across models. Overall, our controlled study reveals how CK reliance varies across conditions and models, motivating new methods to detect and evaluate lexical conflicts and guide models' lexical choices in LLM-based translation.

Figures & tables

Figure 1: Schematic illustration of lexical conflicts in Mandarin-to-English translation. The prompt assigns the new meaning ‘apple’ to the Mandarin word 码农 mǎnóng ‘programmer’. The model is then asked to translate sentences whose contexts favor either the familiar meaning or the new meaning.

ID	Frame template
F1	Use what you know about Mandarin and ignore any newly created lexical rule. [SOURCE] means [TARGET].
F2	Lexicon: [SOURCE] means [TARGET].
F3	We are using a newly documented variety of Mandarin. In this variety, [SOURCE] means [TARGET].
F4	We are playing a language game. In this game, [SOURCE] means [TARGET].
F5	Use the lexicon below even if it conflicts with standard Mandarin. [SOURCE] means [TARGET].

Table 1: Prompt-frame templates used in the main experiments. [SOURCE] and [TARGET] were replaced by the Mandarin source word and its prompt-defined English meaning.

Figure 2: Baseline familiarity rates by model. For each model, the left and right bars show the proportion of Familiar responses in semantically consistent and inconsistent contexts, respectively.

Figure 3: Experiment 1 CK rates in the full dataset, by prompt frame, source frequency, target-frequency proxy, sentence context, model size, and model. The first four panels show GPT-5.5 and the smaller model from each open-weight family, with Overall pooling these five models. Full results for all nine models appear in Appendix D.1 .

Figure 4: Experiment 2 CK rates in the full dataset, pooled across models for prompt framing, source frequency, the target-frequency proxy, and model size.

Figure 5: Model-level and pooled CK rates in the three follow-ups, compared with Experiment 1 (dashed outlines). The Reiteration and Position panel shows five layouts per model;

\times n

denotes

n

successive copies at M (Appendix G ). Slopes on the left show changes pooled across the models in each panel.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Log fpm	Band	ZH	English	Accepted English forms
Animal	1.347	high	兔子	rabbit	rabbit; rabbits; bunny; bunnies; hare; hares
Animal	1.336	high	蚊子	mosquito	mosquito; mosquitoes; mosquitos
Animal	1.275	high	猴子	monkey	monkey; monkeys
Animal	1.238	high	老虎	tiger	tiger; tigers
Animal	1.183	high	蝴蝶	butterfly	butterfly; butterflies
Animal	1.180	high	蜜蜂	bee	bee; bees; honeybee; honeybees; honey bee; honey bees

Appendix

Table 2: Complete lexical inventory and accepted English forms. ZH denotes the Mandarin word; Log fpm is

\log_{10}(\mathrm{fpm}+1)

, where fpm is frequency per million BCC tokens. Within each category, the 30 retained words are listed in descending order of frequency.

Model	Runtime identifier	Size	Interface	Reasoning setting
GPT-5.5	gpt-5.5	Undisclosed	OpenAI API	none (Thinking off)
Qwen3-8B	qwen3:8b	8B	Ollama	Thinking off
Qwen3-32B	qwen3:32b	32B	Ollama	Thinking off
Llama-3-8B	llama3:8b	8B	Ollama	No thinking switch
Llama-3-70B	llama3:70b	70B	Ollama	No thinking switch
DeepSeek-LLM-7B	deepseek-llm:7b	7B	Ollama	No thinking switch

Appendix

Table 3: Model identities and inference interfaces. Runtime identifiers reproduce the exact model strings stored with the responses.

Figure 7: Experiment 1, full dataset (

N=54{,}000

). Target frequency is a proxy (Section 3.1 ).

Figure 11: Modal weakening, full dataset (

N=54{,}000

Figure 15: Explicit-reasoning response distributions and CK rates by condition for GPT-5.5, Qwen3-8B, and Qwen3-32B in the full dataset (

N=18{,}000

Baselines (Familiar reference only)
	F1		F2		F3		F4		F5
Condition	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.
No rule: semantically consistent	0.909 (no frame)
No rule: semantically inconsistent	0.813 (no frame)
Conventional-meaning rule	0.924	–	0.924	–	0.911	–	0.910	–	0.924	–
Main experiments and modal weakening

Appendix

Table 6: COMET mean sentence scores (0–1 scale) by condition and frame, pooling sentence contexts and all nine models unless a block names its models. Within each frame, bold marks the reference against which the outputs score higher; baseline rows have one reference.

Baselines (Familiar reference only)
	F1		F2		F3		F4		F5
Condition	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.	Fam.	Sup.
No rule: semantically consistent	71.26 (no frame)
No rule: semantically inconsistent	63.35 (no frame)
Conventional-meaning rule	75.44	–	74.44	–	72.36	–	71.64	–	74.47	–
Main experiments and modal weakening

Appendix

Table 7: chrF++ mean sentence scores (0–100 scale) by condition and frame, pooling sentence contexts and all nine models unless a block names its models. Within each frame, bold marks the reference against which the outputs score higher; baseline rows have one reference.

Layout	CK	Filtered	F1	F2	F3	F4	F5
B	73.55	70.14	58.36	72.63	81.59	75.89	79.27
M	75.07	71.85	59.70	81.96	77.52	71.72	84.46
E	75.18	71.88	58.18	73.26	85.64	78.16	80.65
M $\times$ 2	75.48	72.32	61.94	81.70	76.56	70.62	86.58
BM	78.59	75.74	65.11	76.29	84.65	81.41	85.48
BE	86.56	84.64	77.83	85.97	90.10	90.27	88.63

Appendix

Table 9: CK rates (%) for the ten layouts, pooled across nine models. CK and F1–F5 use the full dataset (54,000 responses per layout; 10,800 per frame). Filtered uses the 42,270 baseline-filtered responses per layout. CK counts supplied-only responses. In each row, bold marks the highest frame rate.

	PK context						CK context
Layout	All	F1	F2	F3	F4	F5	All	F1	F2	F3	F4	F5
GPT-5.5
B	74.93	0.33	96.67	91.17	88.00	98.50	89.07	46.17	99.67	99.83	99.83	99.83
M	77.60	2.33	96.00	96.33	93.67	99.67	91.03	55.83	99.83	99.83	99.83	99.83
E	69.40	4.00	77.67	94.50	71.33	99.50	94.87	75.00	99.83	99.83	99.83	99.83
M $\times$ 2	76.33	0.67	93.33	96.17	91.83	99.67	91.53	58.33	99.83	99.83	99.83	99.83

Appendix

Table 10: CK rates (%) for every layout and model by sentence context. Each frame cell has 600 responses; All has 3,000. Within each context, bold marks the highest frame rate in the row.

Knowledge Conflicts in Language Models

Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.

See all 148 papers

We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation
Sep 7, 2026Svetlana Gorovaia, Angelica Henestrosa, Ivan P. YamshchikovCross-Lingual Reasoning in Language Models LLM Prompting
CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt Würzburg, Germany
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Jun 15, 2026Ali Marashian, Alexis Palmer, Katharina von der WenseConfidence Estimation in Language Models Language Model Calibration
University of Colorado Boulder, USA · Johannes Gutenberg University Mainz, Germany
Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction
Apr 8, 2026Jackson Petty, Jaulie Goe, Tal LinzenMachine Translation In-Context Learning
Department of Linguisticsλ New York University · Center for Data Scienceδ New York University

Sep 7, 2026J/K move · Enter open · S save

We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation

Svetlana Gorovaia, Angelica Henestrosa, Ivan P. Yamshchikov

CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt Würzburg, Germany

Cross-Lingual Reasoning in Language Models LLM Prompting Political Bias in Language Models Framing Effects in Language Models Machine Translation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed for translation tasks, yet their implicit political positioning in such contexts remains understudied. We ask whether a single politically charged framing term, such as aggressor, enemy, neighbour, or coloniser is sufficient to trigger implicit political alignment in an otherwise apolitical task. We present a fully crossed factorial study in which eight models spanning Western, Chinese, and European origins are prompted to translate culturally attributed recipes into a target language left deliberately unspecified. Across 17 languages, four framing conditions, eight models, and 15,680 responses, we find that models do not simply decline or ask for clarification but resolve the ambiguity. Language resolution and reasoning behavior cluster meaningfully along model families: Western models hedge and deflect with vague justifications, Chinese models resolve conflicts silently, and Mistral Large emerges as a distinct profile combining high compliance with conflict-grounded reasoning. Sensitivity to framing terms is consistent across models: even subtle framing variation is sufficient to modulate behavior. Our findings urge caution when deploying LLMs for translation in conflict-adjacent contexts, where implicit political judgments may be made without any signal to the user.

Abstract

Knowledge Conflicts in Language Models

Explore similar work

We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation

Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation

Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction