Paper ID: 2405.11264

Cross-Language Assessment of Mathematical Capability of ChatGPT

Gargi Sathe, Aneesh Shamraj, Aditya Surve, Nahush Patil, Kumkum Saxena

This paper presents an evaluation of the mathematical capability of ChatGPT across diverse languages like Hindi, Gujarati, and Marathi. ChatGPT, based on GPT-3.5 by OpenAI, has garnered significant attention for its natural language understanding and generation abilities. However, its performance in solving mathematical problems across multiple natural languages remains a comparatively unexplored area, especially in regional Indian languages. In this paper, we explore those capabilities as well as using chain-of-thought prompting to figure out if it increases the accuracy of responses as much as it does in the English language and provide insights into the current limitations.

Submitted: May 18, 2024