To Write or to Automate Linguistic Prompts, That Is the Question

Marina Sánchez-Torrón; Daria Akselrod; Jason Rauchwerk

cs.CLMar 26, 2026

To Write or to Automate Linguistic Prompts, That Is the Question

Authors: Marina Sánchez-Torrón, Daria Akselrod, Jason Rauchwerk

Organizations: Smartling 244 Fifth Avenue Suite 1471 New York, NY 10001

Abstract

LLM performance is highly sensitive to prompt design, yet whether automatic prompt optimization can replace expert prompt engineering in linguistic tasks remains unexplored. We present the first systematic comparison of hand-crafted zero-shot expert prompts, base DSPy signatures, and GEPA-optimized DSPy signatures across translation, terminology insertion, and language quality assessment, evaluating five model configurations. Results are task-dependent. In terminology insertion, optimized and manual prompts produce mostly statistically indistinguishable quality. In translation, each approach wins on different models. In LQA, expert prompts achieve stronger error detection while optimization improves characterization. Across all tasks, GEPA elevates minimal DSPy signatures, and the majority of expert-optimized comparisons show no statistically significant difference. We note that the comparison is asymmetric: GEPA optimization searches programmatically over gold-standard splits, whereas expert prompts require in principle no labeled data, relying instead on domain expertise and iterative refinement.

Figures & tables

		BLEU		HTER		TMR
Model	Prompt	Uni	Dec	Uni	Dec	Uni	Dec
GPT-4.1-mini	Manual	0.48	0.46	0.53	0.56	0.86	0.77
	Base	0.45	0.45	0.55	0.57	0.87	0.81
	Opt. DSPy (best)	0.46 (2,3)	0.49 (1)	0.55 (3)	0.56 (1,3)	0.88 (1)	0.91 (2) ‡
GPT-5.4-mini	Manual	0.50	0.48	0.50	0.55	0.90	0.82
	Base DSPy	0.45	0.45	0.56	0.57	0.85	0.78

Table 1: Summary performance for the Terminology Insertion task, macro-averaged across six target locales (ar, de-DE, es-ES, ru-RU, sv-SE, zh-TW). Uni = unified (single-stage); Dec = decomposed (three-stage) N: 1085 (ar=85, de-DE=200, es-ES=200, ru-RU=200, sv-SE=200, zh-TW=200). Each Opt. DSPy cell reports the score from the best-performing optimization objective for that metric and architecture; superscripts identify which: (1) BLEU, (2) HTER, (3) LLM-based judge. Best value per model and column in bold . ‡

p<0.01

vs. the next-best configuration (paired stratified bootstrap, 1,000 iterations, Holm–Bonferroni corrected per model).

		BLEU		HTER		ChrF3
Model	Prompt	Mean	Median	Mean	Median	Mean	Median
GPT-4.1-mini	Manual	57.93	58.77	30.32	25.73	70.55	70.91
	Base DSPy	58.21	60.15	30.27	25.41	70.59	71.78
	Opt. DSPy	59.33	60.13	29.21	25.79	71.27	72.64
GPT-5.4-mini	Manual	57.63	57.82	31.80	28.03	69.69	69.79
	Base DSPy	55.60	55.19	34.07	29.55	68.88	69.06

Table 2: Summary performance for the Translation task, macro-averaged translation scores across ten target locales. N: 972 (de-DE=140, es-ES=69, fr-FR=133, it-IT=15, ja-JP=164, ko-KR=170, pl-PL=60, ru-RU=50, tr-TR=44, zh-CN=127). Mean and median of BLEU, HTER, and ChrF3. BLEU, ChrF3 = higher is better, HTER = lower is better. Best value per model and column in bold . †

p<0.05

, ‡

p<0.01

vs. the next-best configuration (paired stratified bootstrap, 1,000 iterations, Holm–Bonferroni corrected per model).

		Det F1		Cat F1		Sev F1		MQM $r$
Model	Prompt	Uni	Dec	Uni	Dec	Uni	Dec	Uni	Dec
GPT-4.1-mini	Manual	0.64	0.54	0.36	0.39	0.54 ‡	0.53 ‡	0.15	0.18
	Base DSPy	0.52	0.43	0.39	0.44	0.48	0.52	0.21	0.18
	Opt. DSPy	0.63	0.60 ‡	0.41	0.40	0.46	0.50	0.20	0.16
GPT-5.4-mini	Manual	0.60 ‡	0.48	0.36	0.36	0.45	0.38	0.12	0.08
	Base DSPy	0.53	0.41	0.33	0.36	0.45	0.33	0.15	0.11

Table 3: Summary performance for the LQA task, macro-averaged across six target locales. Uni = unified (single-stage); Dec = decomposed (two-stage). N: 1250 (ar=91, de-DE=353, es-ES=362, ru-RU=80, sv-SE=245, zh-CN=119). Best value per model and column in bold . †

p<0.05

, ‡

p<0.01

vs. the next-best configuration (paired stratified bootstrap, 1,000 iterations, Holm–Bonferroni corrected per model).

Prompt Engineering

Prompt engineering designs and evaluates instructions, context, examples, and learned prompts to steer AI model behavior.

See all 124 papers

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs
Apr 18, 2026Wei-Chi Wu, Sheng-Lun Wei, Hen-Hsen Huang +1Multilingual Language Model Evaluation LLM Prompting
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan
Optimizing LLM Prompt Engineering with DSPy Based Declarative Learning
Apr 6, 2026Shiek Ruksana, Sailesh Kiran Kurra, Thipparthi Sanjay BaradwajLLM Prompting Prompt Optimization
Vasavi College of Engineering, Hyderabad, TS, India · Amazon, Texas, United States of America · Vasavi College of Engineering, Telangana, India
Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis
May 26, 2026Shuzhi Gong, Hechuan WenAutomatic Prompt Optimization LLM Reasoning
The University of Melbourne, Melbourne, Victoria, Australia · The University of Queensland, Brisbane, Queensland, Australia

Apr 18, 2026J/K move · Enter open · S save

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

Wei-Chi Wu, Sheng-Lun Wei, Hen-Hsen Huang, Hsin-Hsi Chen

Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan

Multilingual Language Model Evaluation LLM Prompting Low-Resource Language Processing Prompt Engineering

Open paper AskPDF arXiv

Translation-based prompting is widely used in multilingual LLMs, yet its effectiveness varies across languages and tasks. We evaluate prompting strategies across ten languages of different resource levels and four benchmarks. Our analysis shows that no single strategy is universally optimal. Translation strongly benefits low-resource languages even when translation quality is imperfect, high-resource languages gain little, and prompt-based self-routing underperforms explicit translation. Motivated by these findings, we formulate prompting strategy selection as a learned decision problem and introduce lightweight classifiers that predict whether native or translation-based prompting is optimal for each instance. The classifiers achieve statistically significant improvements over fixed strategies across four benchmarks and generalize to unseen task formats not observed during training. Further analysis reveals that language resource level, rather than translation quality alone, determines when translation is beneficial.

Abstract

Prompt Engineering

Explore similar work

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

Optimizing LLM Prompt Engineering with DSPy Based Declarative Learning

Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis