Transparent AI for Mathematics: Transformer-Based Large Language Models for Mathematical Entity Relationship Extraction with XAI

Tanjim Taharat Aurpa

cs.CLMar 6, 2026

Transparent AI for Mathematics: Transformer-Based Large Language Models for Mathematical Entity Relationship Extraction with XAI

Authors: Tanjim Taharat Aurpa

Organizations: Department of Data Science and Engineering, University of Frontier Technology, Bangladesh, .

Abstract

Mathematical text understanding is a challenging task due to the presence of specialized entities and complex relationships between them. This study formulates mathematical problem interpretation as a Mathematical Entity Relation Extraction (MERE) task, where operands are treated as entities and operators as their relationships. Transformer-based models are applied to automatically extract these relations from mathematical text, with Bidirectional Encoder Representations from Transformers (BERT) achieving the best performance, reaching an accuracy of 99.39%. To enhance transparency and trust in the model's predictions, Explainable Artificial Intelligence (XAI) is incorporated using Shapley Additive Explanations (SHAP). The explainability analysis reveals how specific textual and mathematical features influence relation prediction, providing insights into feature importance and model behavior. By combining transformer-based learning, a task-specific dataset, and explainable modeling, this work offers an effective and interpretable framework for MERE, supporting future applications in automated problem solving, knowledge graph construction, and intelligent educational systems.

Figures & tables

Figure 1: Step-by-step pipeline for constructing the Mathematical Entity–Entity Relation Dataset, illustrating text collection from Bangla_MER and Somikoron datasets, duplicate handling, entity extraction, equation analysis, and manual relation labeling.

Text	Entity 1	Entity 2	Relationship
Eighteen players of the Sri Lankan national cricket team have come to play in Bangladesh. Bangladesh team also has eighteen players. There is total thirty six players in two teams.	Eighteen	Eighteen	Addition
Subtracting fifty from ten is equals forty.	Fifty	Ten	Subtraction
Two thousand multiplied by five is ten thousand.	Two Thousand	Two	Multiplication
The square root of four is two .	Four	Two	Square Root
The factorial value of seven is five thousand and forty.	Seven	Five Thousand Forty	Factorial
Man bought twelve mangoes and divided them equally among three children. Each child got four of mangoes.	Twelve	Three	Division

Table 1: The observations of the developed dataset by combining Bangla_MER and Somikoron Datasets.

Figure 2: Donut chart showing the percentage distribution of mathematical relation classes in the constructed dataset, highlighting the dominance of subtle Square Root operations and the balanced representation of Addition, Subtraction, Multiplication, and Division.

Figure 3: Illustration of the data preprocessing workflow, demonstrating raw text normalization through character cleaning, stop-word removal, lemmatization, and stemming to generate model-ready mathematical text.

Figure 4: Proposed BERT Model

Hyperparameters	BERT
learning rate (AdamW)	2e-04
max_len	50
Batch_size	12
verbose	1
epoch	40

Table 2: This table lists the model parameters and their corresponding values.

Figure 6: Overall workflow of the proposed approach, integrating data collection and preprocessing, transformer-based model training and evaluation, and SHAP-based explainability for transparent mathematical relation extraction.

Figure 7: Comparative performance of transformer-based models (BERT, ELECTRA, RoBERTa, ALBERT, DistilBERT, and XLNet) in terms of accuracy, micro-average F1 score, and macro-average F1 score, demonstrating the superior performance of BERT.

Figure 8: Training and validation loss curves across epochs for the proposed BERT-based model. The figure shows rapid convergence during the initial epochs, followed by stable training loss and a slightly increasing validation loss, indicating effective learning with minor overfitting at later stages.

Figure 9: Confusion matrix heatmap for the BERT model on the test dataset, visualizing correct classifications and misclassifications across six mathematical relation classes.

Class	TP	TN	FP	FN	Precision	Recall	Specificity	Error Rate	F1 Score
Factorial	70	466	0	0	100	100	100	0	100
Addition	99	459	8	1	93	99	98	1.8	96
Subtraction	96	467	0	2	100	98	100	0.4	99
Multiplication	116	449	4	0	97	100	99	0.8	98
Division	101	454	0	4	100	96	100	0.8	98
Square Root	163	392	0	0	100	100	100	0	100

Table 3: Different Evaluation Metrices for the BERT model

Figure 10: SHAP text plots illustrating word-level feature contributions for predicting distinct mathematical relation classes, where red highlights indicate positive contributions and blue highlights indicate negative contributions.

Figure 11: SHAP-based word contribution analysis for the “Division” class across multiple observations, showing how feature importance influences prediction confidence for both relevant and irrelevant samples.

Figure 12: SHAP bar plots presenting mean feature importance values for each mathematical relation class, revealing operation-specific keywords and distributed contextual contributions that drive model predictions.

Relation Extraction

Relation extraction identifies and classifies relationships between entities or events in text and multimodal data.

See all 82 papers

Tracing Mathematical Proficiency Through Problem-Solving Processes
Nov 29, 2025Jungyang Park, Suho Kang, Jaewoo Park +4Intelligent Tutoring Systems Knowledge Tracing
1Yonsei University · 2Mathpresso · 3Seoul National University
DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems
Jan 11, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Sudipta KarNumerical Reasoning in Language Models Mathematical Reasoning Benchmarks
University of California, Riverside · University of Maryland, Baltimore County · Oracle Health AI
SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
Jun 29, 2026Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova +3Text Embedding Evaluation LLM Reranking
INSAIT, Sofia University “St. Kliment Ohridski” · ETH Zurich

Nov 29, 2025J/K move · Enter open · S save

Tracing Mathematical Proficiency Through Problem-Solving Processes

Jungyang Park, Suho Kang, Jaewoo Park, Jaehong Kim+3

1Yonsei University · 2Mathpresso · 3Seoul National University

Intelligent Tutoring Systems Knowledge Tracing

Open paper AskPDF arXiv

Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' Mathematical Proficiency (MP) as intermediate representation. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency. Code is available here.

Abstract

Relation Extraction

Explore similar work

Tracing Mathematical Proficiency Through Problem-Solving Processes

DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems

SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics