MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese
Authors: Tiago Teixeira, Ana Carolina Erthal, Juan Belieni, Beatriz Canaverde, Diego Mesquita, Miguel Faria, Eliezer de Souza da Silva, André F. T. Martins
Organizations: Instituto Superior Técnico, Universidade de Lisboa · Fundação Getulio Vargas · Instituto de Telecomunicações · Universidade de Coimbra, CISUC/LASI, DEI · Basque Center for Applied Mathematics
The use of large language models (LLMs) for complex mathematical reasoning is an emergent area of research, with fast progress in methods, models, and benchmark datasets. However, most mathematical reasoning evaluations exhibit a significant linguistic bias, with the vast majority of benchmark datasets being exclusively in English or (at best) translated from English. We address this limitation by introducing {\sc Math-PT}, a novel dataset comprising 1,729 mathematical problems written in European and Brazilian Portuguese. {\sc Math-PT} is curated from a variety of high-quality native sources, including mathematical Olympiads, competitions, and exams from Portugal and Brazil. We present a comprehensive benchmark of current state-of-the-art LLMs on {\sc Math-PT}, revealing that frontier reasoning models achieve strong performance in multiple choice questions compared to open weight models, but that their performance decreases for questions with figures or open-ended questions. To facilitate future research, we release the benchmark dataset and model outputs.
Figures & tables
Country
Competition
Level
MC
OE
Total
Portugal
OPM
1
78
34
112
2
169
96
265
3
143
193
336
4
29
221
250
Brazil
OBMEP
2
118
–
118
3
140
–
140
Table 1: Number of questions per competition level and type across European and Brazilian Portuguese.
Model
Level
pt-PT
pt-BR
MC
OE
MC
OE
GPT-5
L1
96.15
100.00
N/A
N/A
L2
94.67
89.58
90.68
90.62
L3
96.50
92.75
92.86
91.38
L4
96.55
85.52
93.33
89.47
L5
N/A
N/A
90.00
N/A
Table 2: Performance of several models in multiple choice and open-ended questions and different levels.
Model
Level
MC
OE
No Fig.
Fig.
No Fig.
Fig.
GPT-5
L1
100.00
88.00
100.00
100.00
L2
96.52
90.74
88.46
90.91
L3
97.92
93.62
94.35
89.86
L4
100.00
87.50
83.83
90.74
Qwen3-235B-A22B
L1
98.11
76.00
100.00
73.33
Table 3: Impact of visual content (figures) in pt-PT.
Model
Level
pt-PT
pt-BR
MC
OE
MC
OE
Qwen3-8B
L1
84.62
82.35
N/A
N/A
L2
76.33
58.33
87.29
75.00
L3
78.32
61.14
87.14
72.41
L4
65.52
50.23
81.48
52.63
L5
N/A
N/A
76.36
N/A
Table 4: Impact of model size within the Qwen3 family.
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl +14
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · Independent Researcher +7
Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that models robustness in HRL setting do not necessarily translate to LRL. Moreover, proprietary models, such as Gemini 2.5 Flash and GPT-4.1 are less robust to digit, whereas Gemini 3.0 Pro is more robust. Among open models, GPT-OSS 120B and DeepSeek v3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.
Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro +6
McGill University · Mila-Quebec AI Institute · University of Toronto +7
Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.