cs.CLMar 11, 2026
SaveMultilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environments
Organizations: Hasso Plattner Institute & ELLIS Unit Potsdam · Apple
Abstract
We present the Multilingual Reasoning Gym, an extension of Reasoning Gym (Stojanovski et al., 2025), that procedurally generates verifiable reasoning problems across 14 languages. We translate templates for 94 tasks with native-speaker validation in 10 languages and targeted code or template adaptations to ensure linguistic naturalness. The Multilingual Reasoning Gym preserves the core benefits of the procedural generation approach used in the original Reasoning Gym, such as virtually unlimited problem instance generation and adjustable difficulty, and remains directly usable for Reinforcement Learning from Verifiable Rewards and evaluation settings. Problems in the Multilingual Reasoning Gym are parallel across languages, enabling crosslingually parallel data generation at massive scale due to the procedural nature of the environments. We release our implementation to support research into multilingual reasoning models.
Figures & tables
| English (Original) | Translated (Language) | Details |
| Example 1 – Number Sequence Q: 8, 14, 20, 26, 32, 38, 44, ? A: 50 | Japanese Q: 8、14、20、26 、32、38、44、？ A: 50 | Full-width punctuation translated (comma \fontspec_if_language:nTF ENG \addfontfeature Language=English, -> 、 and question mark \fontspec_if_language:nTF ENG \addfontfeature Language=English? -> ？ ). |
| Example 2 – Greatest Common Divisor Q: Find the Greatest Common Divisor (GCD) of these numbers: 688, 716. Give only the GCD as your final answer. A: 4 | German Q: Bestimme den größten gemeinsamen Teiler (ggT) von diesen Zahlen: 688, 716. Gib nur den ggT als Antwort an. A: 4 | Translated mathematical concepts using appropriate terminology commonly used in the target language (Greatest Common Divisor \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishGCD größter gemeinsamer Teiler \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishggT ). |
| Example 3 – String Isomorphism Q: Return True if the following two strings are isomorphic, or False otherwise: zh lr A: True | Italian Q: Restituisci Vero se le seguenti due stringhe sono isomorfe, altrimenti Falso: zh lr A: Vero | Answer keywords translated ( \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTrue/False \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishVero/Falso ). |
| Example 4 – Bit Count Q: How many 1 bits are there in the binary representation of the number 82789451? A: 14 | Japanese Q: 2進数における数値82789451の1ビットの個数を求めよ。 A: 14 | Uses commonly understood translation for technical terms, e.g., “binary” “2進”. |
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 1: Examples of problems generated from translated templates and the corresponding English originals. Mathematical structure and verifiers remain identical; surface form (punctuation, terminology, answer tokens) is naturally translated.
| Model | en | zh | de | es | fr | it | pt | ru | ja | ko | th | bn | te | sw | Average |
| gemma-3-1b-it | 4.7 | 4.4 | 3.6 | 4.5 | 3.6 | 4.2 | 4.5 | 5.6 | 2.5 | 2.3 | 4.4 | 3.6 | 2.8 | 1.6 | 3.7 |
| gemma-3-4b-it | 16.1 | 16.7 | 17.1 | 20.0 | 18.4 | 19.3 | 19.1 | 17.5 | 15.4 | 16.6 | 17.7 | 14.6 | 15.1 | 12.9 | 16.9 |
| gemma-3-12b-it | 23.2 | 19.7 | 19.1 | 20.9 | 20.6 | 21.4 | 19.8 | 20.1 | 19.2 | 19.5 | 19.7 | 17.5 | 17.7 | 17.7 | 19.7 |
| Qwen3-1.7B | 28.4 | 28.2 | 30.3 | 30.8 | 30.3 | 29.2 | 29.4 | 28.8 | 30.4 | 25.9 | 27.0 | 24.6 | 23.0 | 9.9 | 26.9 |
| Qwen3-4B | 44.8 | 41.1 | 42.7 | 43.3 | 41.7 | 41.3 | 42.8 | 37.1 | 42.7 | 41.3 | 41.5 | 36.2 | 34.8 | 26.2 | 39.8 |
| Qwen3-8B | 45.7 | 43.4 | 46.3 | 45.6 | 45.4 | 45.1 | 44.7 | 41.5 | 46.0 | 43.9 | 43.3 | 39.2 | 37.9 | 32.3 | 42.9 |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 2 : Average@8 scores of different models across languages on our Multilingual Reasoning Gym dataset at easy task difficulty (25th percentile of defined difficulty progressions). We use the recommended sampling parameters by the publisher for each model. Bold indicates best result within each model family. Underlined bold indicates overall best result.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | en | zh | de | es | fr | it | pt | ru | ja | ko | th | bn | te | sw | Average |
| gemma-3-1b-it | 3.5 | 2.7 | 2.9 | 3.6 | 2.4 | 3.3 | 2.9 | 3.7 | 1.5 | 1.2 | 2.2 | 1.9 | 1.7 | 0.6 | 2.4 |
| gemma-3-4b-it | 10.5 | 10.6 | 11.2 | 12.5 | 12.3 | 12.4 | 12.4 | 11.0 | 8.7 | 10.5 | 11.0 | 8.9 | 10.0 | 6.5 | 10.6 |
| gemma-3-12b-it | 14.9 | 12.6 | 12.2 | 14.0 | 12.9 | 13.4 | 12.6 | 12.7 | 11.5 | 12.4 | 12.1 | 10.4 | 11.7 | 11.0 | 12.4 |
| Qwen3-1.7B | 17.3 | 18.0 | 20.3 | 20.0 | 19.4 | 18.5 | 18.4 | 18.4 | 20.3 | 17.3 | 17.4 | 16.1 | 15.2 | 5.3 | 17.3 |
| Qwen3-4B | 31.7 | 29.2 | 30.4 | 30.7 | 29.6 | 29.7 | 30.2 | 26.6 | 29.7 | 28.5 | 29.7 | 25.6 | 25.6 | 18.5 | 28.3 |
| Qwen3-8B | 30.3 | 30.0 | 32.2 | 31.4 | 32.4 | 31.8 | 30.5 | 29.2 | 31.3 | 29.8 | 30.2 | 27.1 | 26.5 | 22.1 | 29.6 |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 3 : Average@8 scores of different models across languages on our Multilingual Reasoning Gym dataset at hard task difficulty (75th percentile of difficulty progression). We use the recommended sampling parameters by the publisher for each model. Bold indicates best result within each model family. Underlined bold indicates overall best result.
| Model | en | zh | de | es | fr | it | pt | ru | ja | ko | th | bn | te | sw | Average |
| gemma-3-1b-it | 7.2 | 7.7 | 5.1 | 6.2 | 6.2 | 5.8 | 7.2 | 7.6 | 4.3 | 4.0 | 6.4 | 6.1 | 4.2 | 2.3 | 5.7 |
| gemma-3-4b-it | 24.4 | 25.6 | 26.5 | 28.1 | 28.5 | 27.9 | 28.1 | 26.8 | 24.2 | 24.7 | 25.9 | 22.0 | 21.6 | 20.1 | 25.3 |
| gemma-3-12b-it | 33.1 | 30.1 | 28.7 | 30.1 | 30.1 | 30.4 | 29.3 | 29.2 | 29.4 | 29.5 | 29.8 | 26.4 | 26.3 | 26.8 | 29.2 |
| Qwen3-1.7B | 41.3 | 38.9 | 42.1 | 42.8 | 43.2 | 41.0 | 41.6 | 40.0 | 43.1 | 36.6 | 38.5 | 31.4 | 30.3 | 12.3 | 37.4 |
| Qwen3-4B | 62.0 | 56.4 | 59.6 | 58.5 | 59.0 | 57.0 | 57.6 | 52.6 | 59.4 | 54.6 | 59.2 | 50.2 | 45.9 | 34.4 | 54.7 |
| Qwen3-8B | 63.1 | 59.7 | 63.9 | 61.4 | 62.1 | 61.7 | 61.4 | 57.1 | 63.0 | 59.7 | 60.3 | 55.4 | 54.3 | 45.8 | 59.2 |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 4 : Average@8 scores of different models across languages on our Multilingual Reasoning Gym dataset at medium task difficulty. We use the recommended sampling parameters by the publisher for each model. Bold indicates best result within each model family. Underlined bold indicates overall best result.
| Model | en | zh | de | es | fr | it | pt | ru | ja | ko | th | bn | te | sw | Average |
| gemma-3-1b-it | 21.2 | 22.3 | 18.9 | 21.5 | 21.1 | 19.3 | 21.7 | 22.8 | 16.2 | 16.4 | 19.4 | 20.1 | 15.1 | 9.3 | 18.9 |
| gemma-3-4b-it | 41.7 | 41.4 | 44.2 | 45.8 | 45.2 | 45.6 | 44.9 | 41.8 | 41.9 | 41.6 | 44.8 | 38.9 | 41.5 | 39.4 | 42.8 |
| gemma-3-12b-it | 54.5 | 49.3 | 51.6 | 53.0 | 51.2 | 52.1 | 51.2 | 49.4 | 50.8 | 51.1 | 51.6 | 45.8 | 47.0 | 49.2 | 50.6 |
| Qwen3-1.7B | 61.4 | 58.6 | 60.5 | 61.8 | 62.1 | 58.7 | 60.3 | 58.7 | 62.6 | 56.2 | 58.3 | 52.6 | 50.5 | 26.6 | 56.4 |
| Qwen3-4B | 75.1 | 72.1 | 74.7 | 74.4 | 74.3 | 73.5 | 73.4 | 70.5 | 74.4 | 71.3 | 74.2 | 68.8 | 63.7 | 52.1 | 70.9 |
| Qwen3-8B | 76.8 | 75.0 | 77.5 | 75.8 | 76.7 | 76.1 | 76.8 | 74.2 | 76.3 | 74.7 | 75.6 | 72.7 | 68.9 | 65.3 | 74.5 |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 5 : Pass@8 scores of different models across languages on our Multilingual Reasoning Gym dataset at medium task difficulty. We use the recommended sampling parameters by the publisher for each model. Bold indicates best result within each model family. Underlined bold indicates overall best result.
| Model | en | zh | de | es | fr | it | pt | ru | ja | ko | th | bn | te | sw | Average |
| gemma-3-1b-it | 82.5 | 5.4 | 25.3 | 17.2 | 23.6 | 24.0 | 16.2 | 23.6 | 12.2 | 13.8 | 18.6 | 24.6 | 21.0 | 10.0 | 22.7 |
| gemma-3-4b-it | 98.2 | 66.2 | 93.8 | 76.1 | 68.7 | 92.7 | 91.1 | 93.9 | 88.8 | 84.1 | 96.4 | 92.3 | 95.6 | 76.8 | 86.8 |
| gemma-3-12b-it † | 0.0 | 8.0 | 30.5 | 50.3 | 28.9 | 27.1 | 0.0 | 26.4 | 18.5 | 12.4 | 27.1 | 25.6 | 29.6 | 22.5 | 21.9 |
| Qwen3-1.7B | 99.9 | 93.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 53.8 | 0.6 | 0.0 | 0.0 | 0.1 | 0.2 | 2.9 | 17.9 |
| Qwen3-4B | 100.0 | 93.7 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 99.4 | 0.1 | 0.0 | 0.1 | 2.1 | 0.4 | 0.1 | 21.1 |
| Qwen3-8B | 100.0 | 93.5 | 0.3 | 0.2 | 0.0 | 0.0 | 0.0 | 78.8 | 0.0 | 0.0 | 0.0 | 1.6 | 0.1 | 1.1 | 19.7 |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 6 : Language consistency scores (average@8) of different models across languages on our Multilingual Reasoning Gym dataset at medium task difficulty. We use the recommended sampling parameters by the publisher for each model. Bold indicates best result within each model family. Underlined bold indicates overall best result. † : language consistency results are skewed for gemma-3-12b-it because the model continued to generate unrelated content after providing answers to the posed question.
| English (Original) | Translated (Language) | Explanation |
| Example 1 – Spell Backward Q: Spell this word backward (example: sun -¿ nus): palpation A: noitaplap | Spanish Q: Escribe esta palabra al revés (ejemplo: sol -¿ los): palpation A: noitaplap | The words are sampled from an English corpus but the task itself does not rely on any knowledge of the English language but rather just requires mechanistically reversing a sequence of letters . Future work could add sufficiently large corpora for each language or use other generation techniques to generate letter sequences. |
| Example 2 – Letter Counting Q: How many times does the letter “w” appear in the text: “it into a watering place”? A: 1 | Japanese Q: テキスト「it into a watering place」の中に文字「w」は何回現れるか？ A: 1 | The sentences are sampled from an English corpus but the task itself does not rely on any knowledge of the English language but rather just requires mechanistically counting characters . Future work could add sufficiently large corpora for each language or use other generation techniques to generate letter sequences. |
| Example 3 – Group Anagrams Q: \fontspec_if_language:nTF ENG \addfontfeature Language=English[...] Group the following list of words into anagrams: \fontspec_if_language:nTF ENG \addfontfeature Language=English["escrod", "decors", "scored", "semitaur", "muriates"] A: \fontspec_if_language:nTF ENG \addfontfeature Language=English[["decors", "escrod", "scored"], ["muriates", "semitaur"]] | German Q: \fontspec_if_language:nTF ENG \addfontfeature Language=English[...] Gruppiere die folgende Wortliste in Anagramme: \fontspec_if_language:nTF ENG \addfontfeature Language=English["escrod", "decors", "scored", "semitaur", "muriates"] A: \fontspec_if_language:nTF ENG \addfontfeature Language=English[["decors", "escrod", "scored"], ["muriates", "semitaur"]] | The anagrams are sampled from an English corpus but the task itself does not rely on any knowledge of the English language but rather just requires mechanistically matching counts of characters . Future work could add sufficiently large corpora for each language, however this is more involved than simply collecting a list of words as building a data-structure of anagrams is required. Additionally, in some languages, the concept of an anagram might not be commonly used (e.g. some non-Latin scripts). |
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 7: Examples of translated tasks which still contain some English language content. We list these separately for easy filtering. Note that for many tasks, the English content is not actually required to be understood semantically but simply serves as an input for string-based algorithmic tasks.
| English (Not translated) | Reason |
| Example 1 – Word ladder Q: Question: Transform the word ladder ’HAND’ to ’GLEE’ by changing one letter at a time. Provide your answer as a comma-separated sequence of uppercase letters without spaces. Each step must be a valid English word. A: \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishHAND,RAND,REND,FEND,FEED,FLED,FLEE,GLEE | This task requires significant knowledge of English, so we cannot use it while keeping the English content as is. However, translation to other languages – especially with non-Latin script – poses a significant challenge, might even be impossible, or significantly alter the difficulty of the task. |
| Example 2 – Time Intervals Q: A database query started at 05/12/2013 and ended at 06/12/2013. How long did the query take? Answer in D days. A: \fontspec_if_language:nTF ENG \addfontfeature Language=English31 days | Some templates in the task require disambiguation between US and European date formatting conventions (without providing such necessary context). In the example, the answer would be “1 day” if interpreted e.g. with European conventions (05/12/2013 -¿ 5th December 2013, 06/12/2013 -¿ 6th December 2013) instead of the ground truth “31 days” if interpreted in an US context (05/12/2013 -¿ 12th May 2013, 06/12/2013 -¿ 12th June 2013). Future work could insert such context or otherwise work on localizing the various date formatting conventions. |
| Example 3 – Figlet Fonts Q: What word does this say? \fontspec_if_language:nTF ENG \addfontfeature Language=English \fontspec_if_language:nTF ENG \addfontfeature Language=English .d88b. 888b. .d88b db .d88b. 8P Y8 8 .8 8P dPYb YPwww. 8b d8 8wwK’ 8b dPwwYb d8 ‘Y88P’ 8 Yb ‘Y88P dP Yb ‘Y88P’ | (1) The words are sampled from an English corpus – localization would require constructing large-scale corpora for each language. This could be tackled in future work via existing dictionaries. (2) However, for languages using non-ASCII characters, the rendering algorithm to the “ASCII-Art” figlet fonts depicted would have to be adapted, which is non-trivial. |
\fontspec_if_language:nTF ENG \addfontfeature Language=EnglishTable 8: Examples of tasks which were not translated due to various issues. Our implementation still supports the original English version.
| Category | Task | Translation Status | Notes |
|---|---|---|---|
| Algebra (6 tasks) | |||
| Algebra | \fontspec_if_language:nTF ENG \addfontfeature Language=Englishcomplex_arithmetic | Fully translated | – |
| Algebra | \fontspec_if_language:nTF ENG \addfontfeature Language=Englishintermediate_integration | Fully translated | – |
| Algebra | \fontspec_if_language:nTF ENG \addfontfeature Language=Englishpolynomial_equations | Fully translated with adaptation | Fixed rounding description error |
| Algebra | \fontspec_if_language:nTF ENG \addfontfeature Language=Englishpolynomial_multiplication | Fully translated | Fixed non-determinism |
| Algebra | \fontspec_if_language:nTF ENG \addfontfeature Language=Englishsimple_equations | Fully translated with adaptation | Fixed whitespace issue |
\fontspec_if_language:nTF ENG \addfontfeature Language=English Table 9 : Complete list of Reasoning Gym tasks and their translation status
More in this topic
Synthetic Data Generation for Language Models
Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.
See all 293 papers