Large-enrollment university courses face persistent challenges in providing timely and scalable instructional support. While generative AI holds promise, its effective use depends on reliability and pedagogical alignment. We present a human-centered case study of AI-assisted support in a Calculus I course, implemented in close collaboration with the course instructor. We developed a system to answer students' questions on a discussion forum, fine-tuning a lightweight language model on 2,588 historical student-instructor interactions. The model achieved 75.3% accuracy on a benchmark of 150 representative questions annotated by five instructors, and in 36% of cases, its responses were rated equal to or better than instructor answers. Post-deployment student survey (N = 105) indicated that students valued the alignment of the responses with the course materials and their immediate availability, while still relying on the instructor verification for trust. We highlight the importance of hybrid human-AI workflows for safe and effective course support.
Figures & tables
Figure 1. Overview of our human-guided research process: (1) collecting and annotating student–instructor Q&A pairs, (2) instructor-led model fine-tuning and evaluation, (3) expert evaluation by five Calculus instructors through surveys and interviews, and (4) post-deployment evaluation using student surveys and instructors’ review of model responses.
Figure 2. Example from the dataset, showing the context, student question, instructor answer, and our model response.
Figure 3. Survey layout showing the information presented to instructors (student question, context, instructor answer, and model response) and the evaluation criteria used for annotation.
Figure 4. Human annotation results for the dataset of 150 questions on (a) Correctness, (b) Relevance, and (c) Completeness. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
Figure 5. Evaluation of base and finetuned models on 40 test questions by a single expert. The results are shown by the question’s difficulty: we have 10 examples of easy questions, 20 medium, and 10 hard ones. Base models: Llama 3.1, Mathstral, Qwen 2.5, Gemma 3, and DeepSeek R1 Distill. Models with “-QA” are finetuned on our dataset; “OpenMath220k” are finetuned on the respective math dataset. Scores shown are averages. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
Metric
Difficulty
Agreement (%)
Correctness
All
92.00
Easy
93.33
Hard
86.67
Medium
95.00
Relevance
All
62.00
Easy
53.33
Table 1. Agreement (%) across annotators by metric and difficulty level.
Figure 6. Distribution of model response alignment across question types for a set of 150 Calculus I questions. Each question type (e.g., Explanation, Hint, Clarification) is labeled with its proportion in the dataset. Bar colors represent the degree of alignment between the model’s response and the student question intent: Misaligned , Partially aligned , or Fully aligned .
Figure 7. Model vs. Human agreement on (a) Correctness, (b) Relevance, and (c) Completeness. The model shown is GPT-o4-mini, and human labels are derived via majority vote.
Figure 8. Model vs. Human agreement on (a) Expectations, (b) Expectation Alignment, and (c) Comparison. The model shown is GPT-o4-mini, and human labels are derived via majority vote.
Figure 9. Comparison across annotators and automated evaluation with GPT-4o-mini. 95% confidence intervals are computed via non-parametric bootstrap with 10000 resamples.
Figure 10. Overview of model statistics: (a) Distribution of instructor actions taken on model-generated answers, and (b) Distribution of question volume by hour of day across weekdays.
Figure 11. Student perceptions of the model’s responses, based on survey ratings (n = 105). (SA=Strongly Agree, A=Agree, N=Neutral, D=Disagree, SD=Strongly Disagree)
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Context
Question
Answer
mean / max
mean / max
mean / max
French
891 / 5320
56 / 651
90 / 725
English
864 / 5214
52 / 634
82 / 692
Appendix
Table 2. Token statistics for context, questions, and non-augmented original answers in French and English datasets based on the GPT-4o tokenizer.
Model
Model Family
Epochs
Context Window
Llama-3.1-8B-Inst
llama
-
128k
Mathstral-7B
mistral
-
32k
Qwen-2.5-Math-7B
qwen
-
4k
OpenR1-Qwen-7B
qwen
-
32k
Gemma-3-12B-Inst
gemma
-
128k
DeepSeek-R1-Distill-14B
qwen
-
128k
Appendix
Table 3. Overview of model configurations. * in the context window column for Open-R1-QA indicates that the base Qwen-2.5-Math-7B model was extended from a 4k to a 32k token context window. The Gemma-QA_OpenMath220k model was fine-tuned for 3 epochs on the OpenMath220k dataset, followed by 3 epochs on the course dataset. All other models were only fine-tuned on course data.
Model
AIME24
AIME25
MATH_500
GPQA:Diamond
U_math
Llama-3.1-8B-Inst
0.066
0.000
0.460
0.369
0.144
Mathstral-7B
0.033
0.066
0.486
0.298
0.146
Qwen-2.5-Math-7B
0.133
0.133
0.834
0.272
0.302
OpenR1-Qwen-7B
0.533
0.400
0.932
0.424
0.504
Gemma-3-12B-Inst
0.267
0.200
0.860
0.349
0.342
DeepSeek-R1-Distill-14B
0.667
0.533
0.958
0.611
0.541
Appendix
Table 4. Performance of the base models and the fine-tuned model on the selected math benchmarks. “QA” means that the model was finetuned on our course data, “OpenMath220k” means that it was also finetuned on the OpenR1-Math-220k dataset. A dash (–) indicates that the model was not evaluated on that metric. We highlight the best results across the base models and our models with bold .
Figure 12. Weekly usage statistics of asked and kept questions
Model
Price / 1M Input Tokens ($)
Price / 1M Output Tokens ($)
Our pipeline
0.050
0.15
GPT-5
1.25
10
GPT-5 mini
0.25
2
GPT-5 nano
0.05
0.40
DeepSeek
0.28
0.42
Gemini 2.5 Pro
1.25
10
Appendix
Table 5. Comparison of model pricing per 1M tokens.
Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from UC Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of free-response quiz submissions from nearly 800 students. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated as correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable, real-course deployment. Building on the dataset and evaluation framework developed here, we outline a standardized benchmark for AI grading of handwritten mathematics to support reproducible comparison and future research.
Zhiqi Yu, Xingping Liu, Haobin Mao +4
UC Irvine Irvine, USA · Independent Shanghai, China · UC Berkeley Berkeley, USA
Timely and specific feedback is one of the strongest influences on student learning, yet it is difficult to sustain in large electrical engineering classes where the ratio of students to demonstrators is high and a learner who is stuck may wait days to find out why an approach was wrong. Generative Artificial Intelligence (GenAI) offers a way to scale conversational feedback, but using it to grade assessed work raises trust and accountability concerns, and keeping a human in the loop to assure its judgements reintroduces the very delay that erodes the value of feedback. The result is a tension between the immediacy that makes feedback so impactful and the human oversight that makes it trustworthy. In this work, we set out to resolve that tension in practice by designing and piloting a GenAI practice platform that delivers immediate, scaffolded feedback during self-directed practice. This relocates human oversight from real-time grading to the upfront verification of solutions. Our goal was to understand how students engaged with the tool, how they perceived the value and reliability of its feedback, and what lessons transfer to other engineering subjects.
Providing timely and accurate learning support in large-scale online coding courses is challenging, particularly in resource-constrained contexts. We present Kwame 2.0, a bilingual (English-French) generative AI teaching assistant built using retrieval-augmented generation and deployed in a human-in-the-loop forum within SuaCode, an introductory mobile-based coding course for learners across Africa. Kwame 2.0 retrieves relevant course materials and generates context-aware responses while encouraging human oversight and community participation. We deployed the system in a 15-month longitudinal study spanning 15 cohorts with 3,717 enrollments across 35 African countries. Evaluation using community feedback and expert ratings shows that Kwame 2.0 provided high-quality and timely support, achieving high accuracy on curriculum-related questions, while human facilitators and peers effectively mitigated errors, particularly for administrative queries. Our findings demonstrate that human-in-the-loop generative AI systems can combine the scalability and speed of AI with the reliability of human support, offering an effective approach to learning assistance for underrepresented populations in resource-constrained settings at scale.
George Boateng, Samuel Boateng, Victor Kumbol
ETH Zurich, Switzerland · Kwame AI Inc., U.S. · Charité - Universitätsmedizin Berlin, Germany