The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
Organizations: Cohere Labs · Cohere
Abstract
Synthetic data has become a cornerstone for scaling LLMs, yet its benefits for multilingual modeling remain bottlenecked by translation. Translation for data augmentation inherits English-centric framing and style and neglects cultural dimensions, ultimately constraining model generalization. We argue that the overlooked prompt space offers a more powerful lever for improving multilingual performance. We introduce a lightweight framework for prompt-space optimization, where translated prompts are systematically transformed for Naturalness, Cultural Adaptation, and Difficulty Enhancement. Using an off-the-shelf multilingual LLM, we apply these transformations to prompts from 12 languages spanning 7 families. Under identical data conditions, our approach achieves substantial and consistent downstream improvements over the translation-only baseline: +4.7% accuracy on Global-MMLU, +2.4% XCometXL on Flores and +35.3% wins in preferences on mArenaHard. These gains are further corroborated by human preferences on open-ended creative writing. We establish prompt-space optimization as a simple yet powerful paradigm for building multilingual LLMs that are more robust, culturally grounded, and globally capable.
Figures & tables
| Length | Rel. Dist. | Perplexity | Diversity | Difficulty | Quality | |||||||
| Transformation | P | C | P | C | P | C | P | C | P | C | P | C |
| Translated | 406 | 2451 | – | – | 15.34 | 2.46 | 0.88 | 0.77 | 1.78 | 1.77 | 3.21 | 4.78 |
| Naturalized | 397 | 2490 | 0.24 | 0.64 | 14.06 | 2.48 | 0.90 | 0.77 | 1.76 | 1.75 | 3.26 | 4.81 |
| Cultural | 470 | 2352 | 0.30 | 0.67 | 12.11 | 2.51 | 0.89 | 0.79 | 1.76 | 1.76 | 3.28 | 4.82 |
| Difficulty | 1936 | 5322 | 0.86 | 0.81 | 3.13 | 2.15 | 0.77 | 0.76 | 2.44 | 2.45 | 4.50 | 4.83 |
| Cultural + Diff. | 1205 | 3873 | 0.58 | 0.74 | 4.52 | 2.27 | 0.82 | 0.77 | 1.97 | 2.10 | 3.75 | 4.76 |
| Prompts in FT | Flores | G-MMLU | Include44 | MGSM | mArenaHard | PolyWrite |
| xCometXL | Accuracy | Win-rate % | ||||
| Translated | 0.786 | 52.5 | 47.0 | 56.2 | – | – |
| Naturalized | 0.791 | 53.1 | 46.9 | 56.5 | 57.7 | 63.8 |
| Cultural | 0.805 | 57.9 | 50.8 | 66.0 | 65.7 | 66.1 |
| Difficulty | 0.816 | 54.5 | 51.2 | 65.1 | 61.8 | 64.6 |
| Cultural + Diff. | 0.810 | 57.2 | 51.8 | 67.3 | 67.7 | 66.9 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Script | Lang. Family | Resourcedness | WMT24++ | Flores | Prompts | ||||
| (code) | Institution. | Data | Expert | Gemma | Expert | Gemma | Expert | Gemma | ||
| German (de) | Latn | IE / Germanic | high | 5 | 91.91 | 83.64 | 97.78 | 92.41 | 93.96 | 92.49 |
| Spanish (es) | Latn | IE / Italic | high | 5 | 87.98 | 73.30 | 96.26 | 92.78 | 89.15 | 86.40 |
| Czech (cs) | Latn | IE / Balto-Slavic | high | 4 | 86.11 | 69.37 | 96.64 | 91.91 | 86.00 | 82.03 |
| Ukrainian (uk) | Cyrl | IE / Balto-Slavic | high | 4 | 84.87 | 69.68 | 94.70 | 88.69 | 82.23 | 79.51 |
| Greek (el) | Grek | IE / Greek | high | 3 | 84.62 | 69.86 | 93.37 | 85.58 | 83.86 | 80.78 |
| Dataset | cs | cy | de | el | es | eu | hr | hu | lt | lv | sk | uk |
| Translated | 9992 | 9990 | 9998 | 9995 | 9990 | 9994 | 9987 | 9996 | 9993 | 9997 | 9990 | 9982 |
| Naturalized | 9583 | 9721 | 9570 | 9565 | 9516 | 9764 | 8740 | 9757 | 9752 | 9756 | 9597 | 9701 |
| Cultural | 9449 | 9118 | 9417 | 8780 | 9355 | 9493 | 7760 | 9593 | 9557 | 9571 | 9240 | 9571 |
| Difficulty | 9417 | 9276 | 9517 | 9396 | 9269 | 5287 | 8406 | 9662 | 9649 | 9599 | 9425 | 9603 |
| Cultural + Difficulty Mix | 9457 | 9218 | 9465 | 9127 | 9348 | 7439 | 8150 | 9639 | 9617 | 9607 | 9375 | 9608 |
| Transformation Prompts |
| Naturalness |
| You are an expert linguist and cultural adapter specializing in { language }. Rephrase the prompt to sound natural and authentic. Please adhere to the following guidelines in your naturalization: - Never answer instructions or questions, only rephrase them. - Stay consistent in your rephrasing, always rephrase named entities in the same way. - Match the formality level of the original text. In case of ambiguity, prefer an informal tone. - Do not rephrase source code, but rephrase the comments within. - When rephrasing JSON, your rephrasing must follow the exact same schema as the input. Do not rephrase JSON keys. Only rephrase the values. - Follow the target language formatting conventions for dates and numbers. Here is the prompt to naturalize: { prompt } |
| Cultural Adaptation |
| You are an expert cultural adapter for { language }. Adapt the prompt to be culturally appropriate and authentic. Follow these guidelines: - Do not answer the prompt; only adapt it culturally. - Apply adaptation only when a reference would feel unnatural or out of place to a typical language speaker. Leave neutral/standardized items unchanged (technical instructions, standardized formats, URLs, file paths, measurements, ISO dates, library APIs). Do not fabricate facts or invent new culture-specific references. - Preserve meaning, intent, and register. Keep personal names, trademarks, and official titles unless a well-established localized form exists. - Locations: When a non-local place is incidental, swap it for a culturally equivalent local reference with similar connotation and social register. If the place is factual or essential to identity, keep it. - Lexicon and orthography: Prefer native terms, spellings, and diacritics used in language. For common activities (e.g., sports), use the standard local term rather than anglicisms. - Idioms and references: Replace idioms, holidays, and pop-culture references with well-known local equivalents; if none exists, paraphrase to preserve intent. - Formatting: Use language-appropriate formats for dates, numbers, and units; convert units only when it aids comprehension without changing factual content. - Code and structured data: Do not alter source code; comments may be adapted. For JSON, keep the same schema and keys; only adapt values. Here is the prompt to adapt: { prompt } |
| Difficulty |
| You are an expert task designer for { language }. Rewrite the prompt to increase its difficulty and complexity while preserving the original intent and domain. Follow these guidelines: - Do not answer the prompt; only rewrite it to be more difficult. - Preserve the original intent and domain. Do not change the topic or introduce factual errors. - Tailor the rewrite to the prompt’s task type (e.g., coding, data wrangling/SQL, creative writing, analysis/explanation, classification, dialogue/roleplay). - Code and JSON: do not alter source code; comments/instructions may be made more demanding. For JSON, keep the same schema and keys; only adapt values and constraints. - Ensure the task remains solvable with the given information; if extra assumptions are needed, require the solver to list and justify them. Here is the prompt to complexify: { prompt } |
| Transformation | Result |
| Seed prompt | What’s the difference between thai ginger and regular ginger? |
| Translation(de) | Was ist der Unterschied zwischen thailändischem Ingwer und normalem Ingwer? |
| Naturalisation | Was ist der Unterschied zwischen thailändischem Ingwer und dem Ingwer, den man normalerweise kennt? |
| Cultural Adaptation | Was ist der Unterschied zwischen thailändischem Ingwer und dem Ingwer, den man hierzulande üblicherweise im Handel findet? |
| Difficulty | Vergleichen und kontrastieren Sie detailliert thailändischen Ingwer (Kha) mit dem in der europäischen Küche üblicherweise verwendeten Ingwer (Zingiber officinale). Ihre Analyse soll folgende Aspekte berücksichtigen: botanische Unterschiede (einschließlich, aber nicht beschränkt auf Rhizom-Morphologie und ätherische Öl-Zusammensetzung), typische Anbaubedingungen und -methoden in Thailand versus anderen Anbauregionen, kulinarische Anwendungen und Geschmacksnuancen (unter Berücksichtigung regionaler Variationen innerhalb Thailands und Europas), sowie traditionelle medizinische Verwendungen in den jeweiligen Kulturen.\n\nIhre Antwort soll in Form eines strukturierten Essays verfasst sein, der mindestens 500 Wörter umfasst. Zitieren Sie mindestens drei wissenschaftliche Quellen (z.B. botanische Datenbanken, ethnobotanische Studien, landwirtschaftliche Forschungsberichte) und zwei Kochbücher oder kulinarische Fachartikel, die sich explizit mit Ingwer befassen.\n\nDarüber hinaus sollen Sie eine Tabelle erstellen, die die wichtigsten Unterschiede in den oben genannten Aspekten übersichtlich zusammenfasst.\n\nSollten Sie Annahmen über spezifische Ingwersorten innerhalb Europas treffen (z.B. hinsichtlich Herkunft oder Anbaumethoden), so sind diese explizit zu benennen und zu begründen. Ebenso sind etwaige Schwierigkeiten bei der Beschaffung spezifischer Informationen zu Kha zu erwähnen und die angewandten Strategien zur Informationsfindung zu erläutern. |
| Seed prompt | Is it possible to have a permanent job and a part-time job at the same time in the UK? |
| Language | Translation | Naturalized | |
| de | 91.50 | 91.11 | -0.39 |
| es | 85.42 | 84.88 | -0.56 |
| cs | 86.21 | 83.27 | -2.95 |
| hu | 84.84 | 85.25 | +0.41 |
| hr | 80.45 | 79.68 | -0.77 |
| uk | 85.34 | 85.10 | -0.24 |
| Name | Language List | Metric |
| GlobalMMLU | cs, en, de, el, lt, es, uk | Accuracy |
| Include44 | eu, hr, de, el, hu, lt, es, uk | Accuracy |
| Flores | eu, hr, cs, de, el, hu, lv, lt, sk, es, uk, cy | xCometXL |
| MGSM++ | eu, cs, en, de, el, hu, es, cy | Accuracy |
| PolyWrite | en, de, es, cs, uk, hu, el, sk, hr, lt, lv, eu, cy | self-BLEU, win rates |
| mArenaHard-v2.0++ | en, de, es, cs, uk, el, lv, lt, hu, hr, sk, cy, eu | win rates |
| Issue | Overall | German | Spanish |
| Grammar | 23 | 13 | 10 |
| Fluency | 19 | 15 | 4 |
| Other | 17 | 8 | 9 |
| Creativity | 15 | 8 | 7 |
| Punctuation | 11 | 6 | 5 |
| Chattiness | 10 | 10 | 0 |
| Self-BLEU | NWR | LPR | |
| Translated | 33.73 | – | 97.6 |
| Naturalized | 30.01 | 57.1 | 97.3 |
| Cultural | 32.65 | 63.7 | 97.5 |
| Difficulty | 34.01 | 69.3 | 97.3 |
| Cultural + Diff. | 29.77 | 66.6 | 97.9 |
| Category | Business | Culture | Health | Other | STEM |
| Translated | 0.524 | 0.524 | 0.320 | 0.396 | 0.410 |
| Naturalized | 0.627 | 0.515 | 0.372 | 0.372 | 0.439 |
| Cultural | 0.534 | 0.568 | 0.372 | 0.399 | 0.482 |
| Difficulty | 0.689 | 0.581 | 0.399 | 0.372 | 0.444 |
| Cult. + Diff. (mix) | 0.588 | 0.587 | 0.393 | 0.411 | 0.447 |
| Model | Cultural-Agnostic (CA) | Cultural-Sensitive (CS) |
| Translated | 0.677 | 0.605 |
| Naturalized | 0.676 | 0.635 |
| Cultural | 0.700 | 0.670 |
| Difficulty | 0.708 | 0.666 |
| Cultural + Difficulty Mix | 0.683 | 0.642 |
| Model | Unsupported | Supported |
| Translated | 0.716 | 0.882 |
| Naturalized | 0.719 | 0.890 |
| Cultural | 0.732 | 0.907 |
| Difficulty | 0.749 | 0.908 |
| Cultural + Difficulty Mix | 0.742 | 0.905 |
| Model | cs | cy | de | el | en | es | eu | hr | hu | lt | lv | sk | uk | Average |
| Naturalized | 0.552 | 0.537* | 0.533* | 0.550 | 0.800 | 0.568 | 0.592 | 0.517* | 0.545 | 0.600 | 0.562 | 0.558 | 0.600 | 0.578 |
| Cultural | 0.623 | 0.606 | 0.646 | 0.629 | 0.846 | 0.700 | 0.622 | 0.636 | 0.640 | 0.666 | 0.602 | 0.601 | 0.723 | 0.657 |
| Difficulty | 0.597 | 0.564 | 0.653 | 0.630 | 0.815 | 0.667 | 0.544* | 0.578 | 0.605 | 0.587 | 0.655 | 0.540* | 0.600 | 0.618 |
| Cult. + Diff. Mix | 0.617 | 0.685 | 0.630 | 0.643 | 0.851 | 0.692 | 0.623 | 0.676 | 0.714 | 0.683 | 0.663 | 0.654 | 0.662 | 0.676 |
| Model | cs | cy | de | el | en | es | eu | hr | hu | lt | lv | sk | uk | Average |
| Naturalized | 0.682 | 0.607 | 0.662 | 0.626 | 0.632 | 0.684 | 0.686 | 0.597 | 0.623 | 0.600 | 0.603 | 0.675 | 0.617 | 0.638 |
| Cultural | 0.692 | 0.613 | 0.701 | 0.619 | 0.594 | 0.774 | 0.677 | 0.649 | 0.672 | 0.665 | 0.626 | 0.630 | 0.682 | 0.661 |
| Difficulty | 0.646 | 0.598 | 0.766 | 0.555* | 0.574* | 0.787 | 0.697 | 0.571* | 0.695 | 0.710 | 0.594 | 0.552* | 0.656 | 0.646 |
| Cultural + Difficulty Mix | 0.591 | 0.668 | 0.740 | 0.568* | 0.742 | 0.755 | 0.677 | 0.649 | 0.682 | 0.684 | 0.677 | 0.623 | 0.643 | 0.669 |
| Model | cs | de | el | en | es | lt | uk | Average |
| Translated | 0.529 | 0.533 | 0.506 | 0.605 | 0.558 | 0.440 | 0.503 | 0.525 |
| Naturalized | 0.535 | 0.546 | 0.507 | 0.602 | 0.567 | 0.450 | 0.510 | 0.531 |
| Cultural | 0.579 | 0.590 | 0.556 | 0.665 | 0.619 | 0.486 | 0.554 | 0.579 |
| Difficulty | 0.555 | 0.535 | 0.540 | 0.575 | 0.565 | 0.495 | 0.555 | 0.545 |
| Cultural + Difficulty Mix | 0.573 | 0.583 | 0.550 | 0.649 | 0.613 | 0.488 | 0.551 | 0.572 |
| Model | de | el | es | eu | hr | hu | lt | uk | Average |
| Translated | 0.489 | 0.409 | 0.544 | 0.358 | 0.556 | 0.373 | 0.489 | 0.542 | 0.470 |
| Naturalized | 0.460 | 0.427 | 0.575 | 0.334 | 0.567 | 0.391 | 0.453 | 0.542 | 0.469 |
| Cultural | 0.496 | 0.447 | 0.613 | 0.346 | 0.620 | 0.405 | 0.530 | 0.609 | 0.508 |
| Difficulty | 0.518 | 0.465 | 0.598 | 0.336 | 0.651 | 0.407 | 0.539 | 0.580 | 0.512 |
| Cultural + Difficulty Mix | 0.482 | 0.480 | 0.644 | 0.372 | 0.644 | 0.405 | 0.530 | 0.584 | 0.518 |
| Model | cs | cy | de | el | es | eu | hr | hu | lt | lv | sk | uk | Average |
| Translated | 0.879 | 0.671 | 0.947 | 0.834 | 0.906 | 0.614 | 0.808 | 0.710 | 0.711 | 0.692 | 0.810 | 0.845 | 0.786 |
| Naturalized | 0.882 | 0.662 | 0.946 | 0.844 | 0.933 | 0.615 | 0.802 | 0.714 | 0.725 | 0.693 | 0.825 | 0.846 | 0.791 |
| Cultural | 0.904 | 0.673 | 0.954 | 0.858 | 0.935 | 0.614 | 0.821 | 0.733 | 0.736 | 0.707 | 0.842 | 0.881 | 0.805 |
| Difficulty | 0.905 | 0.698 | 0.954 | 0.861 | 0.936 | 0.588 | 0.831 | 0.764 | 0.767 | 0.746 | 0.856 | 0.884 | 0.816 |
| Cultural + Difficulty Mix | 0.900 | 0.684 | 0.954 | 0.856 | 0.933 | 0.601 | 0.822 | 0.747 | 0.757 | 0.731 | 0.855 | 0.880 | 0.810 |
| Gemma3-27B | 0.918 | 0.623 | 0.924 | 0.855 | 0.928 | 0.633 | 0.901 | 0.886 | 0.824 | 0.776 | 0.916 | 0.888 | 0.839 |
| Model | cs | cy | de | el | en | es | eu | hu | Average |
| Translated | 0.665 | 0.350 | 0.645 | 0.625 | 0.745 | 0.705 | 0.295 | 0.470 | 0.5625 |
| Naturalized | 0.640 | 0.295 | 0.665 | 0.680 | 0.750 | 0.735 | 0.265 | 0.490 | 0.5650 |
| Cultural | 0.736 | 0.456 | 0.740 | 0.752 | 0.840 | 0.800 | 0.376 | 0.580 | 0.660 |
| Difficulty | 0.750 | 0.385 | 0.775 | 0.780 | 0.810 | 0.790 | 0.320 | 0.595 | 0.6506 |
| Cultural + Difficulty Mix | 0.780 | 0.472 | 0.756 | 0.752 | 0.876 | 0.788 | 0.400 | 0.556 | 0.673 |
| Benchmark | cs | cy | de | el | en | es | eu | hr | hu | lt | lv | sk | uk | Average |
| mArenaHard | 0.427 | 0.794 | 0.438 | 0.696 | 0.226 | 0.421 | 0.766 | 0.578 | 0.603 | 0.660 | 0.648 | 0.568 | 0.556 | 0.568 |
| PolyWrite | 0.942 | 0.794 | 0.890 | 0.994 | 0.284 | 0.768 | 0.974 | 0.974 | 0.987 | 0.981 | 0.948 | 0.987 | 0.974 | 0.884 |
| Completion Grader (Quality & Difficulty Evaluation) |
| You are a strict grader of responses. The questions are enclosed in <question></question> tags, and the answers are enclosed in <answer></answer> tags. Given a question–answer pair, evaluate the <answer> according to the following criteria: |
| - Does the <answer> address the <question> fully in all parts? - Is the <question> logically sound? - Is the <answer> logically sound? - Is the <answer> factually correct and coherent? - Does the <answer> contain any hallucinations? - Is the <answer> properly formatted? - Does the <answer> use correct punctuation for the given language? - Are there any other issues with the <question> or <answer> not covered above? |
| Any <question> or <answer> that fails to meet the above criteria should be penalized accordingly. |
| - Give the question and answer pair a quality rating between [A,B,C,D,E] with A being best and E being worst. The rating should be at the very end and inside the tag "<quality> </quality>" without markdown formatting. - Give the question and answer pair a difficulty rating between [easy, medium, hard]. The rating should come after the quality rating and inside the tag "<difficulty> </difficulty>" without markdown formatting. |
| Prompt Grader (Prompt Quality & Difficulty Evaluation) |
| Judge Prompt |
| Which of the following responses is the most natural-sounding and overall best one for the given instruction in { language }? A good response should follow these rules, with a primary focus on Rule 5 (Naturalness) : - 1) It should be in { language }. - 2) It should complete the request in the instruction. - 3) It should be factually correct and semantically comprehensible. - 4) It should be grammatically correct and fluent. - 5) Crucially, it should sound natural in { language }. This means it uses common phrasing, appropriate tone, and idiomatic expressions (where suitable) that a native speaker would typically use. It should avoid awkward, stilted, or "translated-sounding" sentences. |
| Instruction: { prompt } Response (A): { completion_a } Response (B): { completion_b } |
| FIRST provide a concise comparison of the two responses, evaluating primarily which response sounds more natural and authentic in { language } . Consider if it uses common phrasing and tone typical of a native speaker, while also meeting the other criteria (completeness, correctness, grammar). If one Response is better, explain which you prefer and why, highlighting differences in naturalness. If both responses are identical or equally good or bad (especially in terms of naturalness), explain why. SECOND, on a new line, state exactly one of ’Response (A)’ or ’Response (B)’ or ’TIE’ to indicate your choice of preferred response. |
Multilingual Language Models
Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.
See all 278 papers