Organizations: Department of Computer Science and Engineering University of Moratuwa Moratuwa, Sri Lanka · Department of Electrical and Information Engineering University of Ruhuna Galle, Sri Lanka · Faculty of Information Technology and Communication Sciences Tampere University Tampere, Finland · School of Computing Australian National University Canberra, Australia
Large language models (LLMs) have achieved strong results in mathematical reasoning, and are increasingly deployed as tutoring and learning support tools in educational settings. However, their reliability for students working in non-English languages, especially low-resource languages, remains poorly understood. We examine this gap by evaluating mathematical reasoning in Sinhala and Tamil -- two languages widely used in South Asian schools but underrepresented in artificial intelligence (AI) research. Using a taxonomy of six math problem types, from basic arithmetic to complex unit conflict and optimization problems, we evaluate four prominent large language models. To avoid translation artifacts that confound language ability with translation quality, we construct a parallel dataset in which each problem is independently authored in Sinhala and Tamil by native speakers, and in English by fluent speakers, all with strong mathematical backgrounds. Our analysis demonstrates that while basic arithmetic reasoning transfers robustly across languages, complex reasoning tasks show significant degradation in Tamil and Sinhala. The pattern of failures varies by model and problem type, suggesting that strong performance in English does not guarantee reliable performance across languages. These findings have direct implications for the deployment of AI tools in multilingual classrooms, and highlight the need for language-specific evaluation before adopting large language models as math tutoring aids in non-English educational contexts.
Figures & tables
Fig. 1 : Sample problems from each of the six problem types in English , Sinhala , and Tamil , illustrating the taxonomy’s coverage from basic arithmetic (Type 1) to complex optimization (Type 6).
Fig. 2 : Accuracy (%) across four LLMs, six problem types, and three languages. Darker green indicates higher accuracy; red tones highlight degradation.
Fig. 3 : Radar plots comparing model accuracy (%) across six problem types for each language. Polygon shrinkage from English to Sinhala and Tamil reflects cross-lingual performance loss.
Type
Category
Reasoning Competency
1
Single-Step
Basic arithmetic comprehension
2
Multi-Step
Chained reasoning
3
Extraneous Information
Distractor filtering
4
Unit Conflict
Unit conversion reasoning
5
Logical Deduction
Algebraic formulation
6
Optimization
Constrained optimization
TABLE I : Summary of the six math word problem types and targeted reasoning competencies.
Fig. 4 : Representative example of a unit conflict problem (Type 4) posed to the same LLM (Claude) across three languages. In English, the model correctly identifies that “2 ms” denotes milliseconds and converts accordingly (v = 0.004 m/s). In Sinhala, the model ignores the unit conversion and treats the time as 2 seconds, yielding the incorrect answer of 4 m/s. Tamil produces the same error. This pattern, where reasoning is correct in English, but unit misinterpretation occurs in low-resource languages, illustrates that cross-lingual degradation manifests not as arithmetic failure but as a failure to parse linguistically embedded unit information. Screenshots reproduced for research purposes.
Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, enabling us to localize and compare model parameters that support mathematical reasoning across languages. We find that the extracted math-associated parameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in intermediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower-resource languages reveal smaller sets of relevant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with systematic language-dependent differences.
Behzad Shomali, Luisa Victor, Tim Selbach +5
Lamarr Institute · University of Bonn · Fraunhofer IAIS
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl +14
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · Independent Researcher +7
Large reasoning models (LRMs) achieve strong mathematical reasoning performance in English, but remain much less reliable in many low- and medium-resource languages. This gap is often explained as a failure to understand non-English problem statements. We show that this view is incomplete: even when the problem is given in English, controlling the model's reasoning language can substantially reduce accuracy, suggesting that language also affects reasoning execution itself. To study this effect, we introduce DATG, a Directed Acyclic Trace Graph framework that maps reasoning traces to language-independent mathematical anchors and dependencies. This allows us to align target-language traces with reference DAGs and measure whether they cover required mathematical nodes, respect dependency edges, and avoid harmful mathematical actions. Experiments on the Qwen3 series across 12 languages show that non-English reasoning often suffers from reduced anchor coverage and weaker dependency fidelity, especially in low-resource languages. Motivated by this diagnosis, we propose Loop-Retry and Formula-Retry, two simple test-time controls targeting DATG-exposed failure modes, and show that they consistently improve target-language reasoning performance in low-resource languages.