Investigating the Influence of Prompt and Response Languages on LLM Content Generation

Thi Thanh Nhan Nguyen; Mai Khoi Tieu; Michael A. Riegler; Pål Halvorsen; Thu Nguyen

cs.CLAug 22, 2026

Investigating the Influence of Prompt and Response Languages on LLM Content Generation

Authors: Thi Thanh Nhan Nguyen, Mai Khoi Tieu, Michael A. Riegler, Pål Halvorsen, Thu Nguyen

Organizations: Université de Technologie de Compiègne, Compiègne, Hauts-de-France, France · Norwegian University of Science and Technology (NTNU), Trondheim, Trøndelag, Norway · SimulaMet, Oslo, Norway · Faculty of Information Technology, HUTECH University, Ho Chi Minh City, Vietnam

Abstract

This study examines how prompt and response language influence the behavior of large language models. Using five models, we evaluated answers to 68 non translation questions across four language conditions: English to English, English to Norwegian, Norwegian to Norwegian, and Norwegian to English. After removing refused items, the dataset contains 1348 responses. We measure length differences with Cohen d, semantic fidelity with LabSE cosine similarity, and cross lingual keyword overlap with both raw and soft Jaccard. Prompt language has a strong effect on response length. With English output, Norwegian prompts shorten responses by about thirty seven percent. With Norwegian output, English prompts shorten responses by about forty one percent. The largest cross lingual contrast shows a reduction in word count but a smaller reduction in tokens, reflecting tokenizer differences. Despite variation in length, semantic similarity remains high, and soft Jaccard reveals substantial conceptual overlap that raw Jaccard does not capture. Effect sizes vary across models, indicating heterogeneity. Prompt language is not neutral and systematically shapes output length and lexical realization, with implications for multilingual prompt design.

Figures & tables

Figure 1: Mean word count (95% bootstrap CI) by model and domain on the balanced dataset ( $n=1{,}348$ ).

Model	Mean $ee$	Mean $en$	$d$	Interpretation
GPT-4o	542.5	157.1	4.42	very large
DeepSeek-V3	440.9	175.3	3.11	very large
Gemini 2.5 Pro	661.3	345.8	2.18	very large
Claude 3.5 Haiku	170.9	126.2	1.52	large
Phi-4-multimodal-instruct	324.0	213.5	0.78	medium-to-large

Table 1: Per-model Cohen’s

d

for the

ee

–

en

length gap on the 68 non-translation questions.

n=68

per condition per model.

Figure 2: Word-count distributions by condition on the balanced dataset ( $n=337$ per cell).

	Response: English	Response: Norwegian
Prompt: English	427.8 ( $ee$ )	203.8 ( $en$ )
Prompt: Norwegian	268.2 ( $ne$ )	344.6 ( $nn$ )

Table 2: Mean word count by prompt

\times

response language on the balanced dataset (

n=337

per cell across five models and four domains).

Figure 3: Mean semantic similarity (left) and mean raw-Jaccard keyword overlap (right) by model and domain, averaged over the six pairwise combinations of $\{ee,en,nn,ne\}$ .

Figure 4: Left: distribution of chunk-averaged LaBSE cosine similarity across questions. Right: distribution of ee $\leftrightarrow$ en keyword overlap under raw Jaccard (red) and LaBSE soft Jaccard (green). Raw Jaccard collapses near zero because English and Norwegian keyword surface forms rarely match; soft Jaccard reveals substantial concept-level overlap ( $\approx 0.53$ mean).

Figure 5: Mean information density (95% bootstrap CI) by model and domain under each condition. CIs replace the earlier $\pm$ SD bars so the intervals stay in the non-negative range.

Figure 6: Information-density distributions by condition. Left: mean $\pm$ 95% bootstrap CI. Right: full distribution.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$nn$	$ne$
Claude 3.5 Haiku	2	2
DeepSeek-V3	3	3
Gemini 2.5 Pro	2	2
Phi-4-multimodal-instruct	3	3
GPT-4o	3	3

Appendix

Table 3: Number of missing responses per model

\times

condition.

Model	$ee$	$en$	$nn$	$ne$	Total
Claude 3.5 Haiku	2	7	11	1	21
DeepSeek-V3	0	0	13	0	13
Gemini 2.5 Pro	3	1	1	5	10
Phi-4-multimodal-instruct	2	10	21	14	47
GPT-4o	7	0	12	0	19
Total	14	18	58	20	110

Appendix

Table 4: Number of responses terminating without terminal punctuation, by model and condition.

Term	$\hat{\beta}_{\log}$	SE	$z$	Back-transformed	Direct-means
Intercept	5.320	0.036	146.20	—	—
Prompt: Norwegian	$-0.443$	0.030	$-14.91$	$-36\%$ (ee $\to$ ne)	$-37\%$
Response: Norwegian	$-0.764$	0.030	$-25.79$	$-53\%$ (ee $\to$ en)	$-52\%$
Prompt $\times$ Response (NO,NO)	$+0.966$	0.042	$+23.01$	—	—

Appendix

Table 5: Mixed-effects length model on

\log(1+\text{words})

, analyzed set (

n=1{,}354

;

\text{words}\sim\text{prompt}\times\text{response}+\text{model}+(1\mid\text{question})

). Percentages are back-transformed as

1-\exp(\hat{\beta})

and represent the marginal effect at the reference level.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 431 papers

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs
Apr 18, 2026Wei-Chi Wu, Sheng-Lun Wei, Hen-Hsen Huang +1Multilingual Language Model Evaluation LLM Prompting
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models
Sep 28, 2026Yusheng Zhou, Eleanor Lin, David JurgensPrompt Sensitivity Language Model Generation Evaluation
Electrical Engineering and Computer Science Department University of Michigan Ann Arbor, MI 48105, USA

Apr 18, 2026J/K move · Enter open · S save

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

Wei-Chi Wu, Sheng-Lun Wei, Hen-Hsen Huang, Hsin-Hsi Chen

Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan

Multilingual Language Model Evaluation LLM Prompting Low-Resource Language Processing Prompt Engineering

Open paper AskPDF arXiv

Translation-based prompting is widely used in multilingual LLMs, yet its effectiveness varies across languages and tasks. We evaluate prompting strategies across ten languages of different resource levels and four benchmarks. Our analysis shows that no single strategy is universally optimal. Translation strongly benefits low-resource languages even when translation quality is imperfect, high-resource languages gain little, and prompt-based self-routing underperforms explicit translation. Motivated by these findings, we formulate prompting strategy selection as a learned decision problem and introduce lightweight classifiers that predict whether native or translation-based prompting is optimal for each instance. The classifiers achieve statistically significant improvements over fixed strategies across four benchmarks and generalize to unseen task formats not observed during training. Further analysis reveals that language resource level, rather than translation quality alone, determines when translation is beneficial.

Abstract

LLM Prompting

Explore similar work

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models