cs.CYMar 9, 2026
SaveAI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AI
Organizations: École polytechnique fédérale de Lausanne Switzerland
Abstract
Large-enrollment university courses face persistent challenges in providing timely and scalable instructional support. While generative AI holds promise, its effective use depends on reliability and pedagogical alignment. We present a human-centered case study of AI-assisted support in a Calculus I course, implemented in close collaboration with the course instructor. We developed a system to answer students' questions on a discussion forum, fine-tuning a lightweight language model on 2,588 historical student-instructor interactions. The model achieved 75.3% accuracy on a benchmark of 150 representative questions annotated by five instructors, and in 36% of cases, its responses were rated equal to or better than instructor answers. Post-deployment student survey (N = 105) indicated that students valued the alignment of the responses with the course materials and their immediate availability, while still relying on the instructor verification for trust. We highlight the importance of hybrid human-AI workflows for safe and effective course support.
Figures & tables
Figure 1. Overview of our human-guided research process: (1) collecting and annotating student–instructor Q&A pairs, (2) instructor-led model fine-tuning and evaluation, (3) expert evaluation by five Calculus instructors through surveys and interviews, and (4) post-deployment evaluation using student surveys and instructors’ review of model responses.
Figure 2. Example from the dataset, showing the context, student question, instructor answer, and our model response.
Figure 3. Survey layout showing the information presented to instructors (student question, context, instructor answer, and model response) and the evaluation criteria used for annotation.
Figure 4. Human annotation results for the dataset of 150 questions on (a) Correctness, (b) Relevance, and (c) Completeness. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
Figure 5. Evaluation of base and finetuned models on 40 test questions by a single expert. The results are shown by the question’s difficulty: we have 10 examples of easy questions, 20 medium, and 10 hard ones. Base models: Llama 3.1, Mathstral, Qwen 2.5, Gemma 3, and DeepSeek R1 Distill. Models with “-QA” are finetuned on our dataset; “OpenMath220k” are finetuned on the respective math dataset. Scores shown are averages. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
| Metric | Difficulty | Agreement (%) |
|---|---|---|
| Correctness | All | 92.00 |
| Easy | 93.33 | |
| Hard | 86.67 | |
| Medium | 95.00 | |
| Relevance | All | 62.00 |
| Easy | 53.33 |
Table 1. Agreement (%) across annotators by metric and difficulty level.
Figure 6. Distribution of model response alignment across question types for a set of 150 Calculus I questions. Each question type (e.g., Explanation, Hint, Clarification) is labeled with its proportion in the dataset. Bar colors represent the degree of alignment between the model’s response and the student question intent: Misaligned , Partially aligned , or Fully aligned .
Figure 7. Model vs. Human agreement on (a) Correctness, (b) Relevance, and (c) Completeness. The model shown is GPT-o4-mini, and human labels are derived via majority vote.
Figure 8. Model vs. Human agreement on (a) Expectations, (b) Expectation Alignment, and (c) Comparison. The model shown is GPT-o4-mini, and human labels are derived via majority vote.
Figure 9. Comparison across annotators and automated evaluation with GPT-4o-mini. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
Figure 10. Overview of model statistics: (a) Distribution of instructor actions taken on model-generated answers, and (b) Distribution of question volume by hour of day across weekdays.
Figure 11. Student perceptions of the model’s responses, based on survey ratings (n = 105). (SA=Strongly Agree, A=Agree, N=Neutral, D=Disagree, SD=Strongly Disagree)
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Context | Question | Answer |
|---|---|---|---|
| mean / max | mean / max | mean / max | |
| French | 891 / 5320 | 56 / 651 | 90 / 725 |
| English | 864 / 5214 | 52 / 634 | 82 / 692 |
Table 2. Token statistics for context, questions, and non-augmented original answers in French and English datasets based on the GPT-4o tokenizer.
| Model | Model Family | Epochs | Context Window |
|---|---|---|---|
| Llama-3.1-8B-Inst | llama | - | 128k |
| Mathstral-7B | mistral | - | 32k |
| Qwen-2.5-Math-7B | qwen | - | 4k |
| OpenR1-Qwen-7B | qwen | - | 32k |
| Gemma-3-12B-Inst | gemma | - | 128k |
| DeepSeek-R1-Distill-14B | qwen | - | 128k |
Table 3. Overview of model configurations. * in the context window column for Open-R1-QA indicates that the base Qwen-2.5-Math-7B model was extended from a 4k to a 32k token context window. The Gemma-QA_OpenMath220k model was fine-tuned for 3 epochs on the OpenMath220k dataset, followed by 3 epochs on the course dataset. All other models were only fine-tuned on course data.
| Model | AIME24 | AIME25 | MATH_500 | GPQA:Diamond | U_math |
|---|---|---|---|---|---|
| Llama-3.1-8B-Inst | 0.066 | 0.000 | 0.460 | 0.369 | 0.144 |
| Mathstral-7B | 0.033 | 0.066 | 0.486 | 0.298 | 0.146 |
| Qwen-2.5-Math-7B | 0.133 | 0.133 | 0.834 | 0.272 | 0.302 |
| OpenR1-Qwen-7B | 0.533 | 0.400 | 0.932 | 0.424 | 0.504 |
| Gemma-3-12B-Inst | 0.267 | 0.200 | 0.860 | 0.349 | 0.342 |
| DeepSeek-R1-Distill-14B | 0.667 | 0.533 | 0.958 | 0.611 | 0.541 |
Table 4. Performance of the base models and the fine-tuned model on the selected math benchmarks. “QA” means that the model was finetuned on our course data, “OpenMath220k” means that it was also finetuned on the OpenR1-Math-220k dataset. A dash (–) indicates that the model was not evaluated on that metric. We highlight the best results across the base models and our models with bold .
Figure 12. Weekly usage statistics of asked and kept questions
| Model | Price / 1M Input Tokens ($) | Price / 1M Output Tokens ($) |
|---|---|---|
| Our pipeline | 0.050 | 0.15 |
| GPT-5 | 1.25 | 10 |
| GPT-5 mini | 0.25 | 2 |
| GPT-5 nano | 0.05 | 0.40 |
| DeepSeek | 0.28 | 0.42 |
| Gemini 2.5 Pro | 1.25 | 10 |
Table 5. Comparison of model pricing per 1M tokens.
Figure 21. Student survey. Part 5.